Detection method and device, vehicle control method and device, vehicle and computer storage medium
By identifying and analyzing the reflected light from the point light source within the occlusion range, the problem that autonomous vehicles cannot obtain the status of red and green traffic indicators is solved, and traffic light status detection is realized under occlusion, improving the safety and control of the vehicle accuracy.
Patent Information
- Application Number
- CN202310524187.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-12
AI Technical Summary
The autonomous vehicle cannot obtain the status of the red and green traffic indicator lights when the adjacent vehicles in front are blocked, which affects the safety and control accuracy of the vehicle.
Ambient images are obtained through visual acquisition equipment, reflect light reflected by point light sources within the occlusion range, and semantic analysis is performed to determine the color information of the red and green traffic indicator light, so as to detect the red and green traffic indicator lights without direct viewing. state.
It realizes the accurate detection of the status of red and green traffic indicators when the pass is not possible, improves the driving safety and adaptability of autonomous vehicles, and reduces the risk of system functions missing.
Smart Images

Figure CN118942070A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of autonomous driving technology, and more particularly to a detection and vehicle control method, device, vehicle, and computer storage medium. Background Art
[0002] The current autonomous driving technology routes can be divided into two categories. One is the autonomous driving solution based on beyond-horizon sensors (such as high-precision map data) as a priori information, and the other is the autonomous driving solution based on the light map and heavy perception technology route. The most important difference between the two is that the former is a multi-source sensor data fusion solution based on high-precision maps, which relies heavily on high-precision map data. However, high-precision maps have shortcomings from acquisition to application, such as high production costs, long data production cycles, and untimely data updates, which greatly restrict the implementation of autonomous driving technology. Therefore, through a light map solution, using visual acquisition equipment (such as cameras) to obtain data, by improving vision-based algorithms, based on semantic information, the reconstruction of the real world is achieved, so that the solution can be deployed at the vehicle terminal to support the realization and implementation of autonomous driving functions.
[0003] In this solution, since the autonomous driving relies on visual acquisition equipment, the autonomous driving vehicle will follow the adjacent vehicle in front. Since the adjacent vehicle in front is too high, the line of sight between the front camera of the vehicle and the red and green traffic lights at the intersection is blocked, resulting in the inability to form a line of sight between the front camera of the vehicle and the red and green traffic lights at the intersection. In this case, since the state of the red and green traffic lights in front cannot be effectively obtained, it is difficult for the autonomous driving vehicle to be accurately controlled, affecting the safety of the autonomous driving vehicle. Summary of the invention
[0004] In view of this, an embodiment of the present application provides a detection solution to at least partially solve the above-mentioned problem.
[0005] According to a first aspect of an embodiment of the present application, a detection method is provided, including: when there is an obstruction blocking the line of sight between a visual acquisition device and an observed object, obtaining an image to be processed including environmental information within a first set range in front; based on the image to be processed, determining whether there is reflected light from a point light source on a target object within a second set range of the obstruction; if there is reflected light from the point light source, performing semantic analysis on at least one target object in the image to be processed, and determining color information of the reflected light from the point light source belonging to the observed object based on the result of the semantic analysis.
[0006] According to a second aspect of an embodiment of the present application, a vehicle control method is provided, comprising: determining whether there is an obstruction blocking the line of sight between a visual acquisition device and an observed object, wherein the observed object includes a red and green traffic light; if there is an obstruction blocking the line of sight, using the above-mentioned method to obtain the indication state of the observed object; and determining the control instructions of the vehicle according to the indication state of the observed object.
[0007] According to a third aspect of an embodiment of the present application, a detection device is provided, including: an acquisition module, used to acquire an image to be processed when there is an obstruction blocking the line of sight between a visual acquisition device and an observed object; a detection module, used to determine, based on the image to be processed, whether there is reflected light from a point light source on a target object within a set range of the obstruction; and an analysis module, used to perform semantic analysis on at least one target object in the image to be processed if the reflected light from the point light source exists, and determine color information of the reflected light from the point light source belonging to the observed object based on the result of the semantic analysis.
[0008] According to a fourth aspect of an embodiment of the present application, a vehicle control device is provided, comprising: a first determination module, used to determine whether there is an obstruction blocking the line of sight between a visual acquisition device and an observed object, wherein the observed object includes a red and green traffic light; a second determination module, used to use the device of the third aspect above to obtain the indication state of the observed object if there is an obstruction blocking the line of sight; and a third determination module, used to determine a control instruction of the vehicle according to the indication state of the observed object.
[0009] According to a fifth aspect of an embodiment of the present application, a controller is provided for executing the method of the first or second aspect described above.
[0010] According to the sixth aspect of an embodiment of the present application, a vehicle is provided, comprising a collection device, a controller, a memory, a communication interface and a communication bus, wherein the collection device, the controller, the memory and the communication interface communicate with each other via the communication bus; the collection device is used to collect environmental data of the environment in which the vehicle is located and generate perception data; the memory is used to store at least one executable instruction, and the controller executes the executable instruction based on the perception data to implement the method of the first or second aspect.
[0011] According to a seventh aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the method described in the first or second aspect.
[0012] According to the solution provided in the embodiments of the present application, when the visual acquisition device cannot see the observed object through line of sight, by acquiring the image to be processed and finding the target object with reflected light from the point light source in the image to be processed (that is, the target object that may reflect the light emitted by the red and green traffic lights), and then performing semantic analysis on the target object and processing the semantic analysis results, the state of the red and green traffic lights can be determined by the color of the light of the red and green traffic lights reflected by the target object, thereby realizing the state detection of the red and green traffic lights in the case of no line of sight. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0014] Figure 1A This is a schematic diagram of blocking visibility according to Embodiment 1 of the present application;
[0015] Figure 1B A flowchart of a detection method according to Embodiment 1 of the present application;
[0016] Figure 2A This is a flow chart of the steps of a detection method according to Embodiment 2 of the present application;
[0017] Figure 2B Schematic diagram of the sub-step flow of step S204 according to the second embodiment of the present application;
[0018] Figure 2C Schematic diagram of the sub-step flow of step S206 according to the second embodiment of the present application;
[0019] Figure 3 A schematic diagram of a process flow of a vehicle control method according to Embodiment 3 of the present application;
[0020] Figure 4 is a structural block diagram of a detection device according to Embodiment 4 of the present application;
[0021] Figure 5 is a structural block diagram of a vehicle control device according to Embodiment 5 of the present application;
[0022] Figure 6 It is a schematic diagram of the structure of an electronic device of a vehicle according to the sixth embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0024] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.
[0025] Embodiment 1
[0026] In this embodiment, the method can be applied to the automatic driving process of the vehicle. When the visual acquisition equipment carried by the vehicle cannot see the red and green traffic lights at the intersection, the state of the red and green traffic lights (such as red light, yellow light, or green light) can be identified, thereby providing a basis for the control decision of the automatic driving of the vehicle.
[0027] like Figure 1A As shown, the visible state can be understood as a state in which the visual acquisition device can directly acquire images containing red and green traffic lights. That is to say, there is no obstruction between the visual acquisition device and the red and green traffic lights that blocks the red and green traffic lights and prevents the visual acquisition device from capturing the red and green traffic lights. This is the visible state.
[0028] An example of a state of lack of visibility is, for example, that there is a high obstruction such as a bus (or truck) in front of the autonomous vehicle, so that the visual acquisition device (such as a camera) on the autonomous vehicle cannot capture the red and green traffic lights at the intersection. Since the visual acquisition device cannot capture the red and green traffic lights, the autonomous vehicle cannot determine whether the intersection is passable, making it impossible for the autonomous vehicle to control and drive normally.
[0029] In order to solve this problem, the present application provides a detection method, such as Figure 1B As shown, the method comprises the following steps:
[0030] Step S102: When there is an obstruction that blocks the line of sight between the visual acquisition device and the observed object, an image to be processed including environmental information within a first set range in front is obtained.
[0031] The observed object may be a red or green traffic light.
[0032] For example, when the autonomous driving vehicle is within a certain distance from or within an intersection, the autonomous driving vehicle may need to know the status of the red and green traffic lights at the intersection to determine whether it can pass through the intersection. If it is determined at this time that there is an obstruction between the visual acquisition device and the observed object, the image to be processed can be acquired by the visual acquisition device. The image to be processed includes environmental information within a first set range in front of the autonomous driving vehicle in the direction of travel. The first set range can be determined based on the structure of the visual acquisition device, such as the viewing angle, focal length, etc. of the visual acquisition device. In some examples, images within a 120° range in the forward horizontal direction can be acquired.
[0033] Step S104: Based on the image to be processed, determine whether there is reflected light from a point light source on the target object within the second set range of the occluder.
[0034] The obstruction may be a large truck, bus or other vehicle or object located in front of the autonomous vehicle (i.e., the vehicle itself), whose height may block the visual acquisition device from seeing the red and green traffic lights. Although the visual acquisition device cannot see the red and green traffic lights, some target objects (such as vehicles or other objects) on both sides of the obstruction may reflect the light of the red and green traffic lights, so that there will be point light source reflection on these target objects.
[0035] In order to determine whether there is an object reflecting the light of the red and green traffic lights, the image to be processed can be processed to identify the target object within the second set range of the occlusion in the image to be processed, and the target object can be processed to determine whether there is a point light source reflecting light on the target object.
[0036] In some examples, a trained neural network model (such as a CNN model) can be used to process the image to be processed, so as to determine the coordinates of the occluder and the coordinates and categories of other objects in the image to be processed. Based on the coordinates and categories of each other object, an object that meets the conditions (such as being a vehicle and being within a second set range) can be determined as a target object.
[0037] The aforementioned second setting range can be determined as needed, for example, 10 meters, 20 meters or 50 meters, etc., and there is no limitation on this.
[0038] For the identified target object, it can be determined whether there is reflected light from a point light source on it by analyzing the corresponding image area. If there is reflected light from a point light source, it means that it may reflect the light emitted by the red and green traffic lights, so step S106 can be executed; if there is no reflected light from a point light source, the target object can be processed later.
[0039] Step S106: If the reflected light of the point light source exists, semantic analysis is performed on at least one target object in the image to be processed, and color information of the reflected light of the point light source belonging to the observed object is determined according to the result of the semantic analysis.
[0040] Since the light in the real environment is more complex, the target object may reflect light from more than one light source. In addition, there may be a light source on the target object itself. In order to ensure that the colors of red and green traffic lights can be accurately identified, the identified target object can be semantically analyzed to obtain a semantic analysis result. The semantic analysis result can be used to indicate the different parts of the target object (such as at least one of the vehicle side window glass, the vehicle side door, and the vehicle side frame), and then based on the semantic analysis result, determine the color information of the reflected light from the point light source belonging to the observed object.
[0041] In the present embodiment, when the visual acquisition device cannot see through the observed object, the image to be processed is acquired, and the target object with reflected light from the point light source (that is, the target object that may reflect the light emitted by the red and green traffic lights) is found from the image to be processed, and then semantic analysis is performed on the target object and the results of the semantic analysis are processed, so as to determine the state of the red and green traffic lights by the color of the light of the red and green traffic lights reflected by the target object, thereby realizing the state detection of the red and green traffic lights in the case of no visible line of sight.
[0042] Embodiment 2
[0043] Reference Figure 2A , shows a schematic flow chart of the steps of the detection method of the second embodiment.
[0044] like Figure 2A As shown, the method comprises the following steps:
[0045] Step S202: When there is an obstruction that blocks the line of sight between the visual acquisition device and the observed object, an image to be processed including environmental information within a first set range in front is obtained.
[0046] Taking an autonomous driving vehicle as an example, it can detect the environment through one or more sensors (such as visual acquisition equipment, lidar, etc.) during driving, and then determine whether the autonomous driving vehicle has driven to the vicinity of the intersection based on the detection results, and whether there are any obstructions (such as large trucks or buses, etc.) blocking the line of sight between the visual acquisition equipment and the observed object (such as red and green traffic lights). If it is determined that there are obstructions, the state of the red and green traffic lights can be determined by the method of this embodiment, so that the autonomous driving vehicle can determine whether it can pass through the intersection based on the state. This can improve the driving safety of the autonomous driving vehicle in situations where there is no line of sight, and can also improve the adaptability of the autonomous driving vehicle.
[0047] There may be many ways to detect whether there is an obstruction that blocks visibility, such as capturing an image through a visual acquisition device and determining whether there is an obstruction by identifying the image, but the method is not limited thereto.
[0048] When it is determined that there is an obstruction, a visual acquisition device with a horizontal viewing angle of 120° can be used to collect a forward image to be processed of the autonomous driving vehicle. The image to be processed contains environmental information within a first set range in front of the autonomous driving vehicle.
[0049] Step S204: Based on the image to be processed, determine whether there is reflected light from a point light source on the target object within the second set range of the occluder.
[0050] In some examples, such as Figure 2B As shown, step S204 includes the following sub-steps:
[0051] Sub-step S2041: performing image recognition on the image to be processed to obtain image data of a target object in the image to be processed that is within a second set range of the obstruction.
[0052] For example, the trained neural network model is used to perform image recognition on the image to be processed to identify the information of the obstruction in the image to be processed (such as the coordinates, type, and image data of the obstruction), and the information of other objects (such as the coordinates, type, and image data of the object). For the information of other objects identified, objects of vehicle type and within the second set range can be screened out as target objects, and the image data corresponding to these target objects can be determined. The second set range can be a range with a radius of 20 meters centered on the obstruction, but is not limited thereto.
[0053] Sub-step S2042: performing grayscale analysis on the image data of the target object to determine whether there is reflected light from a point light source on the target object.
[0054] Grayscale analysis can be performed on the image data of each target object, such as converting the image data into grayscale values, and comparing the grayscale values with a preset grayscale range to determine whether there is reflected light from a point light source in the image data of the target object. It should be noted that when comparing, if the grayscale value matches within the preset grayscale range, it can be considered that there is reflected light from a point light source. In this way, it can be determined whether the light source of the red and green traffic lights forms or exists diffuse reflection or specular reflection light of a point light source on the surfaces that may form diffuse reflection or specular reflection, such as the body or glass of an adjacent vehicle on both sides of an obstruction.
[0055] If there is reflected light from a point light source, step S206 is executed.
[0056] Step S206: If the reflected light of the point light source exists, semantic analysis is performed on at least one target object in the image to be processed, and color information of the reflected light of the point light source belonging to the observed object is determined according to the result of the semantic analysis.
[0057] like Figure 2C As shown, in some examples, if the reflected light of the point light source exists, semantic analysis of at least one target object in the image to be processed may include the following sub-steps:
[0058] Sub-step S2061: If there is reflected light from the point light source, use the trained neural network model to perform semantic analysis on the image data of the target object to obtain image data of a selected area in the target object.
[0059] The neural network model may be a CNN model, but is not limited thereto. The image data of the target object is processed by the neural network model to determine the range of each selected area in the target object, thereby determining the corresponding image data.
[0060] The selected area may include at least one of the vehicle side window glass, the vehicle side door and the vehicle side frame, but is not limited thereto. Any area that satisfies certain conditions for light reflectivity can be used as the selected area. Semantic analysis can not only screen out areas with better light reflectivity in the target object, but also remove areas that can emit light by themselves (such as taillights, brake lights, etc.), thereby reducing interference.
[0061] Sub-step S2062: performing point light source reflected light filtering on the image data of the selected area to remove the point light source reflected light outside the observed object.
[0062] For example, if the observed object is a red or green traffic light, the color of the light emitted by it is one of red, yellow, and green. Therefore, if there is reflected light of point light sources of other colors in the image data of the selected area, the reflected light of these point light sources can be filtered out, thereby reducing interference and subsequent calculations. The image data of the area that may include reflected light of point light sources belonging to the observed object after screening can be used as the result of semantic analysis. For example, the reflected light of non-red or green traffic lights (such as non-red, non-yellow, and non-green light) is removed, and the reflected light of the light emitted by its light source (such as red, yellow, and green light) is retained.
[0063] In some examples, the image data of the selected area can be converted into grayscale values, and the grayscale values can be compared with the grayscale value range of preset colors, so as to remove the area corresponding to the reflected light of the point light source that is not within the grayscale value range of the preset color (the area can be the entire selected area or a part of the selected area, without limitation).
[0064] The image data of the area where the remaining point light sources reflect light can be used as a semantic analysis result to facilitate subsequent color analysis.
[0065] For example, determining the color information of the reflected light of the point light source belonging to the observed object according to the semantic analysis result includes the following sub-steps:
[0066] Sub-step S2063: performing color analysis on the semantically analyzed image data of the area containing the reflected light of the point light source belonging to the observed object.
[0067] In some feasible methods, the image data of the area where the point light source reflects light in the semantic analysis result can be input into a trained neural network model (such as a CNN model), and color analysis is performed on it through the model, and its output is used as the color analysis result.
[0068] Alternatively, in some other feasible ways, the image data of the area where the point light source reflects light may be converted into grayscale values, and the grayscale values may be analyzed and the analysis result may be used as the color analysis result.
[0069] Sub-step S2064: Determine the color information of the reflected light of the point light source of the observed object according to the color analysis result.
[0070] The color indicated by the color analysis result is used as the color information of the light reflected by the point light source of the observed object, and then the state of the observed object is determined based on the color information, such as whether it is a red light state, a yellow light state or a green light state, and this is used as information for control decisions of the autonomous driving vehicle.
[0071] Step S208: Determine the automatic driving instruction based on the color information.
[0072] The color information can be transmitted to the autonomous driving decision module to determine the autonomous driving instructions based on the color information.
[0073] The solution of this embodiment is based on computer vision technology. When the autonomous driving vehicle is about to pass through an intersection, the state of the red and green traffic lights is determined by detecting and analyzing the light reflected by the red and green traffic lights on objects around the intersection (such as vehicle windows), thereby inputting longitudinal stop or go decision and control information for the autonomous driving vehicle.
[0074] This method solves the problem in actual driving scenarios where the autonomous driving vehicle follows the adjacent vehicle in front, and the line of sight between the front camera of the vehicle and the red and green traffic lights at the intersection is blocked due to the height of the adjacent vehicle in front. In this case, if the front vehicle moves forward normally according to the indicator light, and the vehicle continues to move forward and has not crossed the stop line, the front camera of the vehicle cannot directly capture the image containing the red and green traffic lights, and thus cannot determine the status of the red and green traffic lights. The autonomous driving system of the vehicle cannot accurately make control instructions to continue moving forward or stop before the stop line, resulting in the loss of function of the autonomous driving control system.
[0075] In order to solve this problem, the solution of this application uses a visual acquisition device (such as a camera) to obtain images of the reflective surfaces (such as vehicle glass or body) of vehicles or other objects on both sides of the high obstacle in front, and then analyzes the image to determine the color of the light of the reflected red and green traffic lights, and then judges whether the signal state of the red and green traffic lights at this time is red or green based on the color, thereby providing a reference for perception input information for the vehicle to stop or move forward, and further realizing the control of the vehicle by fusing the perception results. This improves the safety and efficiency of vehicles passing through intersections and can play an effective supplementary role.
[0076] This method is a method of detecting visible light based on a camera-based visual sensor. It is a technical solution route that captures reflected light signals from camera hardware, analyzes image information, and determines color information. Compared with the solution of obtaining red and green traffic light information through multi-vehicle networking, this single-vehicle intelligent solution does not require communication protocols, a sufficient number of vehicles, or infrastructure as a prerequisite (such as signal synchronization between traffic lights and infrastructure, and broadcasting traffic light status information to vehicles passing through intersections through infrastructure). It is more applicable and has lower costs.
[0077] By further mining and utilizing the vehicle's surrounding environment information and providing more redundant perception information for autonomous driving, the system's perception capabilities can be continuously improved, thereby improving the functional safety of the autonomous driving system.
[0078] Embodiment 3
[0079] Reference Figure 3 , shows a schematic flow chart of the steps of a vehicle control method according to Embodiment 3 of the present application.
[0080] The method comprises the following steps:
[0081] Step S302: Determine whether there is an obstruction that blocks visibility between the visual acquisition device and the observed object, where the observed object includes a red and green traffic light.
[0082] When the visual acquisition device cannot directly capture the observed object, it can be considered that the line of sight is blocked. One way to confirm whether there is an obstruction is, for example, to identify the image captured by the visual acquisition device, and determine whether the line of sight is possible by identifying whether the observed object exists in the image, or to determine whether there is an obstruction by identifying the object in front, without limitation.
[0083] If there is an obstruction that blocks the view, step S304 is executed.
[0084] Step S304: If there is an obstruction that blocks the view, a detection method is used to obtain the indication status of the observed object.
[0085] When there is an obstruction, the color information of the red and green traffic light can be determined by the detection method of the above-mentioned embodiment one or two, and then its indication status can be determined according to the color information, such as allowing passage (green light), not allowing passage (red light), or about to not allow passage (yellow light).
[0086] Step S306: Determine a vehicle control instruction according to the indicated state of the observed object.
[0087] Based on the detected indication state of the observed object, the control instructions of the autonomous driving vehicle are determined, such as moving forward or stopping before the stop line.
[0088] In this way, when the red and green traffic lights are blocked, the indication status of the red and green traffic lights can be indirectly detected through the visual acquisition equipment, thereby providing a basis for the decision-making of the automatic driving control to achieve vehicle control.
[0089] Embodiment 4
[0090] Reference Figure 4 , shows a structural block diagram of a detection device according to embodiment 4 of the present application.
[0091] The detection device comprises:
[0092] An acquisition module 402 is used to acquire an image to be processed when there is an obstruction that blocks the line of sight between the visual acquisition device and the observed object;
[0093] A detection module 404 is used to determine whether there is reflected light from a point light source on a target object within a set range of an occluder based on the image to be processed;
[0094] The analysis module 406 is used to perform semantic analysis on at least one target object in the image to be processed if the reflected light from the point light source exists, and determine the color information of the reflected light from the point light source belonging to the observed object according to the result of the semantic analysis.
[0095] The observed object may be a red or green traffic light.
[0096] For example, when the autonomous driving vehicle is within a certain distance from or within an intersection, the autonomous driving vehicle may need to know the status of the red and green traffic lights at the intersection to determine whether it can pass through the intersection. If it is determined at this time that there is an obstruction between the visual acquisition device and the observed object, the image to be processed can be acquired by the visual acquisition device. The image to be processed includes environmental information within a first set range in front of the autonomous driving vehicle in the direction of travel. The first set range can be determined based on the structure of the visual acquisition device, such as the viewing angle, focal length, etc. of the visual acquisition device. In some examples, images within a 120° range in the forward horizontal direction can be acquired.
[0097] The obstruction may be a large truck, bus or other vehicle or object located in front of the autonomous vehicle (i.e., the vehicle itself), whose height may block the visual acquisition device from seeing the red and green traffic lights. Although the visual acquisition device cannot see the red and green traffic lights, some target objects (such as vehicles or other objects) on both sides of the obstruction may reflect the light of the red and green traffic lights, so that there will be point light source reflection on these target objects.
[0098] In order to determine whether there is an object reflecting the light of the red and green traffic lights, the image to be processed can be processed to identify the target object within the second set range of the occlusion in the image to be processed, and the target object can be processed to determine whether there is a point light source reflecting light on the target object.
[0099] In some examples, a trained neural network model (such as a CNN model) can be used to process the image to be processed, so as to determine the coordinates of the occluder and the coordinates and categories of other objects in the image to be processed. Based on the coordinates and categories of each other object, an object that meets the conditions (such as being a vehicle and being within a second set range) can be determined as a target object.
[0100] The aforementioned second setting range can be determined as needed, for example, 10 meters, 20 meters or 50 meters, etc., and there is no limitation on this.
[0101] For the identified target object, it can be determined whether there is reflected light from a point light source on it by analyzing the corresponding image area. If there is reflected light from a point light source, it means that it may reflect the light emitted by the red and green traffic lights, so the following process can be performed; if there is no reflected light from a point light source, the target object can be processed later.
[0102] Since the light in the real environment is more complex, the target object may reflect light from more than one light source. In addition, there may be a light source on the target object itself. In order to ensure that the colors of red and green traffic lights can be accurately identified, the identified target object can be semantically analyzed to obtain a semantic analysis result. The semantic analysis result can be used to indicate the different parts of the target object (such as at least one of the vehicle side window glass, the vehicle side door, and the vehicle side frame), and then based on the semantic analysis result, determine the color information of the reflected light from the point light source belonging to the observed object.
[0103] Optionally, the detection module 404 is used to perform image recognition on the image to be processed to obtain image data of a target object in the image to be processed that is within a second set range of the obstruction; and perform grayscale analysis on the image data of the target object to determine whether there is reflected light from a point light source on the target object.
[0104] For example, the trained neural network model is used to perform image recognition on the image to be processed to identify the information of the obstruction in the image to be processed (such as the coordinates, type, and image data of the obstruction), and the information of other objects (such as the coordinates, type, and image data of the object). For the information of other objects identified, objects of vehicle type and within the second set range can be screened out as target objects, and the image data corresponding to these target objects can be determined. The second set range can be a range with a radius of 20 meters centered on the obstruction, but is not limited thereto.
[0105] Grayscale analysis can be performed on the image data of each target object, such as converting the image data into grayscale values, and comparing the grayscale values with a preset grayscale range to determine whether there is reflected light from a point light source in the image data of the target object. It should be noted that when comparing, if the grayscale value matches within the preset grayscale range, it can be considered that there is reflected light from a point light source. In this way, it can be determined whether the light source of the red and green traffic lights forms or exists diffuse reflection or specular reflection light of a point light source on the surfaces that may form diffuse reflection or specular reflection, such as the body or glass of an adjacent vehicle on both sides of an obstruction.
[0106] Optionally, the analysis module 406 is used to perform semantic analysis on at least one target object in the image to be processed if the point light source reflected light exists. If the point light source reflected light exists, then use the trained neural network model to perform semantic analysis on the image data of the target object to obtain image data of a selected area in the target object; perform point light source reflected light filtering on the image data of the selected area to remove point light source reflected light outside the observed object, and use the image data of the area of filtered point light source reflected light as the result of the semantic analysis.
[0107] The neural network model may be a CNN model, but is not limited thereto. The image data of the target object is processed by the neural network model to determine the range of each selected area in the target object, thereby determining the corresponding image data.
[0108] The selected area may include at least one of the vehicle side window glass, the vehicle side door and the vehicle side frame, but is not limited thereto. Any area that satisfies certain conditions for light reflectivity can be used as the selected area. Semantic analysis can not only screen out areas with better light reflectivity in the target object, but also remove areas that can emit light by themselves (such as taillights, brake lights, etc.), thereby reducing interference.
[0109] For example, if the observed object is a red or green traffic light, the color of the light emitted by it is one of red, yellow, and green. Therefore, if there is reflected light of point light sources of other colors in the image data of the selected area, the reflected light of these point light sources can be filtered out, thereby reducing interference and subsequent calculations. The image data of the area that may include reflected light of point light sources belonging to the observed object after screening can be used as the result of semantic analysis. For example, the reflected light of non-red or green traffic lights (such as non-red, non-yellow, and non-green light) is removed, and the reflected light of the light emitted by its light source (such as red, yellow, and green light) is retained.
[0110] In some examples, the image data of the selected area can be converted into grayscale values, and the grayscale values can be compared with the grayscale value range of preset colors, so as to remove the area corresponding to the reflected light of the point light source that is not within the grayscale value range of the preset color (the area can be the entire selected area or a part of the selected area, without limitation).
[0111] The image data of the area where the remaining point light sources reflect light can be used as a semantic analysis result to facilitate subsequent color analysis.
[0112] Optionally, the analysis module 406 is used to perform color analysis on the image data of the area after semantic analysis that contains the reflected light from the point light source belonging to the observed object when determining the color information of the reflected light from the point light source belonging to the observed object based on the semantic analysis result; and determine the color information of the reflected light from the point light source belonging to the observed object based on the color analysis result.
[0113] For example, in some feasible methods, the image data of the area where the point light source reflects light in the semantic analysis result can be input into a trained neural network model (such as a CNN model), and color analysis is performed on it through the model, and its output is used as the color analysis result.
[0114] Alternatively, in some other feasible ways, the image data of the area where the point light source reflects light may be converted into grayscale values, and the grayscale values may be analyzed and the analysis result may be used as the color analysis result.
[0115] The color indicated by the color analysis result is used as the color information of the light reflected by the point light source of the observed object, and then the state of the observed object is determined based on the color information, such as whether it is a red light state, a yellow light state or a green light state, and this is used as information for control decisions of the autonomous driving vehicle.
[0116] The detection device of this embodiment is used to implement the corresponding detection methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the detection device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here.
[0117] Embodiment 5
[0118] Reference Figure 5 , shows a structural block diagram of a vehicle control device according to embodiment five of the present application.
[0119] A first determination module 502 is used to determine whether there is an obstruction that blocks the view between the visual acquisition device and the observed object, wherein the observed object includes a red and green traffic light;
[0120] A second determination module 504 is used to obtain the indication state of the observed object using the above-mentioned device if there is an obstruction that blocks the view;
[0121] The third determination module 506 is used to determine the control instruction of the vehicle according to the indicated state of the observed object.
[0122] When the visual acquisition device cannot directly capture the observed object, it can be considered that the line of sight is blocked. A way to confirm whether there is an obstruction is, for example, to identify the image captured by the visual acquisition device, and determine whether the line of sight is possible by identifying whether there is the observed object in the image, or to determine whether there is an obstruction by identifying the object in front, without limitation.
[0123] When there is an obstruction, the color information of the red and green traffic light can be determined by the detection method of the above-mentioned embodiment one or two, and then its indication status can be determined according to the color information, such as allowing passage (green light), not allowing passage (red light), or about to not allow passage (yellow light).
[0124] Based on the detected indication state of the observed object, the control instructions of the autonomous driving vehicle are determined, such as moving forward or stopping before the stop line.
[0125] In this way, when the red and green traffic lights are blocked, the indication status of the red and green traffic lights can be indirectly detected through the visual acquisition equipment, thereby providing a basis for the decision-making of autonomous driving control.
[0126] Embodiment 6
[0127] Reference Figure 6 , showing a schematic structural diagram of a vehicle according to embodiment six of the present application.
[0128] The vehicle of this embodiment includes: a collection device 602, a controller 604, a memory 610, a communication interface 612 and a communication bus, wherein the collection device 602, the controller 604, the memory 610 and the communication interface 612 communicate with each other through the communication bus. The communication interface 612 is used to communicate with other electronic devices or servers.
[0129] The collection device 602 is used to collect environmental data and generate perception data. Exemplarily, the collection device 602 can be implemented as a camera.
[0130] The memory 610 is used to store at least one executable instruction.
[0131] The controller 604 is used to execute the aforementioned executable instructions to implement the aforementioned method, for example, to perform detection based on the image to be processed. Optionally, the controller 604 can perform detection based on the image to be processed. The specific implementation of the above detection can be implemented with reference to the relevant description in the aforementioned embodiment 1 or 2, which will not be repeated here. Exemplarily, the controller 604 can be implemented as a vehicle-mounted embedded chip.
[0132] Optionally, the controller 604 can also make an autonomous driving decision based on the detection results; and, based on the decision results, drive the vehicle to perform driving task processing.
[0133] Generally, the vehicle also includes a driving device 606 and a driving control device 608. Then, after the controller 604 determines the corresponding automatic driving decision, it will send a driving signal to the driving device 606. The driving device 606 uses the driving signal to control the driving control device 608 (such as brakes, accelerators, steering wheels, etc.) in the vehicle to perform corresponding control operations on the automatic driving settings, thereby realizing autonomous driving of the vehicle.
[0134] For the convenience and brevity of description, the specific working process of the vehicle described above can refer to the corresponding process description in the aforementioned method embodiment, and will not be repeated here.
[0135] It should be noted that in the embodiments of the present application, the vehicle may be a vehicle device or a robot device having the following functions:
[0136] (1) Passenger-carrying function, such as family cars and buses;
[0137] (2) Cargo-carrying functions, such as ordinary trucks, van trucks, trailer trucks, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures, etc.;
[0138] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol cars, cranes, hoists, excavators, bulldozers, forklifts, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinklers, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.;
[0139] (4) Entertainment functions, such as entertainment vehicles, amusement park self-driving devices, balance vehicles, etc.;
[0140] (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, engineering rescue vehicles, etc.
[0141] An embodiment of the present application further provides a computer storage medium on which a computer program is stored. When the program is executed by a controller, the detection method described in any one of the above-mentioned method embodiments is implemented.
[0142] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.
[0143] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated controller or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a controller, a microcontroller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a controller or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.
[0144] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.
[0145] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.
Claims
1. A detection method, It is characterized in that include: When there is an obstruction that blocks the line of sight between the visual acquisition device and the observed object, an image to be processed including environmental information within a first set range in front is obtained; Based on the image to be processed, determining whether there is reflected light from a point light source on a target object within a second set range of the occluder; If the reflected light of the point light source exists, a semantic analysis is performed on at least one target object in the image to be processed, and color information of the reflected light of the point light source belonging to the observed object is determined according to the result of the semantic analysis.
2. The method according to claim 1, It is characterized in that The determining, based on the image to be processed, whether there is reflected light from a point light source on the target object within the second set range of the occluder includes: Performing image recognition on the image to be processed to obtain image data of a target object in the image to be processed that is located within a second set range of the obstruction; Grayscale analysis is performed on the image data of the target object to determine whether there is reflected light from a point light source on the target object.
3. The method according to claim 1, It is characterized in that If the reflected light from the point light source exists, semantic analysis is performed on at least one target object in the image to be processed, including: If the reflected light of the point light source exists, semantic analysis is performed on the image data of the target object using the trained neural network model to obtain image data of a selected area in the target object; The image data of the selected area is subjected to point light source reflected light filtering to remove the point light source reflected light outside the observed object, and the image data of the area of the point light source reflected light after filtering is used as the result of semantic analysis.
4. The method according to claim 3, It is characterized in that The selected area includes at least one of a vehicle side window glass, a vehicle side door, and a vehicle side frame.
5. The method according to claim 3, It is characterized in that Determining the color information of the reflected light from the point light source of the observed object according to the semantic analysis result includes: Performing color analysis on the semantically analyzed image data of the region containing the reflected light of the point light source belonging to the observed object; According to the color analysis result, the color information of the reflected light of the point light source belonging to the observed object is determined.
6. A vehicle control method, It is characterized in that include: Determine whether there is an obstruction that blocks the line of sight between the visual acquisition device and the observed object, wherein the observed object includes a red and green traffic light; If there is an obstruction that blocks the view, obtaining the indication state of the observed object using the method described in any one of claims 1 to 5; A control instruction for the vehicle is determined according to the indicated state of the observed object.
7. A detection device, It is characterized in that include: An acquisition module, used to acquire an image to be processed when there is an obstruction that blocks the line of sight between the visual acquisition device and the observed object; A detection module, used for determining whether there is reflected light from a point light source on a target object within a set range of an obstruction based on the image to be processed; The analysis module is used to perform semantic analysis on at least one target object in the image to be processed if the reflected light from the point light source exists, and determine the color information of the reflected light from the point light source belonging to the observed object based on the semantic analysis result.
8. A vehicle control device, It is characterized in that include: A first determination module is used to determine whether there is an obstruction that blocks the line of sight between the visual acquisition device and the observed object, wherein the observed object includes a red and green traffic light; A second determination module, configured to obtain the indication state of the observed object using the device of claim 7 if there is an obstruction that blocks the view; The third determination module is used to determine the control instruction of the vehicle according to the indicated state of the observed object.
9. A controller, It is characterized in that Used to execute the method according to any one of claims 1 to 6.
10. A vehicle, It is characterized in that It includes a collection device, a controller, a memory, a communication interface and a communication bus, wherein the collection device, the controller, the memory and the communication interface communicate with each other through the communication bus; The collection device is used to collect environmental data of the environment where the vehicle is located and generate perception data; The memory is used to store at least one executable instruction, and the controller executes the executable instruction based on the perception data to implement the method as described in any one of claims 1-6.
11. A computer storage medium having a computer program stored thereon, It is characterized in that This is achieved when the program is executed by the controller.
Citation Information
Patent Citations
Method and device for detecting concealed objects in road traffic
CN109703555A
Signal lamp state estimation method and device
CN110660256A
Vehicle control device, autonomous driving vehicle, and vehicle control method
CN115535000A
Method and device for obtaining information about an object in a non-visible, upcoming surrounding area of a motor vehicle
DE102014224762A1
Visual interface display method and apparatus, electronic device, and storage medium
WO2021227520A1
