Target object state identification method, apparatus and device, and computer storage medium

Abstract

The invention provides a target object state recognition method, device and equipment and a computer storage medium, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring environment data acquired when a vehicle runs along a set route at the current moment, wherein the environment data at least comprises point cloud data and image frames of a target object at the current moment; target triads matched with the point cloud data are determined in a plurality of triads of the reference set, and each triad comprises sample images, sample point clouds and the positions of the target objects in the sample images, of the obtained target objects when the vehicle runs along a set route at the historical moment; obtaining the position of a target object in a sample image of the target triple in the sample image; determining a target area in the image frame based on the position of the target object in the sample image; and determining the state of the target object at the current moment based on the pixel values of the pixel points in the target area. According to the invention, the display state of the target object can be accurately determined at low cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a method, device, equipment and computer storage medium for identifying the state of a target object. Background Art

[0002] In the field of rail transit, especially in the field of intelligent driving, accurate detection and status recognition of signal lights are key links in ensuring the safe operation of trains.

[0003] At present, the recognition of traffic light status is mainly achieved by manually collecting a large amount of training data and then conducting model training to achieve image target detection and classification.

[0004] However, the current method requires manual collection of a large amount of training data while ensuring recognition accuracy. The more training data collected, the higher the recognition accuracy, which results in a large amount of cost investment. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, equipment and computer storage medium for identifying the state of a target object, which can accurately identify the state of a target object with low cost investment.

[0006] In a first aspect, an embodiment of the present invention provides a method for identifying a state of a target object, the method comprising: Acquire environmental data collected when the vehicle is traveling along a set route at the current moment, wherein the environmental data at least includes point cloud data and image frames of the target object at the current moment; Determining a target triplet matching the point cloud data in a pre-constructed benchmark set, wherein the benchmark set includes at least one triplet, each triplet including a sample image of a target object, a sample point cloud, and a position of the target object in the sample image acquired when the vehicle travels along the set route at a historical moment; Obtaining the position of the target object in the sample image of the target triplet; Based on the position, determining a target area in the image frame; Based on the pixel values ​​of the pixels in the target area, the state of the target object at the current moment is determined. In an achievable embodiment, the method further includes: Acquire sensor data acquired when the vehicle travels along the set route at a historical moment, the sensor data comprising at least one original point cloud and at least one original image; Selecting an image containing the target object from at least one original image as the sample image; Acquire the location area of ​​the target object in the sample image; According to the timestamp of the sample image, acquiring, in at least one original point cloud, a target point cloud collected at the same time as the sample image as the sample point cloud; The triplet is constructed based on the sample image, the location area where the target object is located in the sample image, and the sample point cloud.

[0007] In another feasible implementation, the obtaining of sensor data acquired when the vehicle travels along the set route at a historical moment includes: acquiring the original image through a camera provided on the vehicle; and acquiring the original point cloud through a radar sensor and an inertial measurement sensor provided on the vehicle.

[0008] In another achievable implementation, the selecting an image containing the target object from at least one original image as the sample image includes: Removing images that do not meet a set condition from all the acquired original images to obtain remaining images, wherein the set condition includes at least one of not meeting a set size and a distance between adjacent original images not meeting a specified threshold; An image containing the target object is selected from the remaining images as the sample image.

[0009] In another possible implementation, the method further includes: Acquire an image captured when the vehicle is at a set distance threshold from the target object; Determine the area occupied by the target object in the image; The set size is determined based on the area of ​​the region.

[0010] In another achievable implementation, determining a target triplet matching the point cloud data in a pre-built benchmark set includes: Divide the sample point cloud into N cubes; Dividing the point cloud data into M cubes; When the degree of overlap between the N cubes and the M cubes is greater than an overlap threshold, taking the sample point cloud as a candidate sample point cloud; Obtaining a first feature point in each candidate sample point cloud and a second feature point in the point cloud data; When the feature matching degree between the first feature point and the second feature point is greater than a matching degree threshold, taking the candidate sample point cloud as a target sample point cloud; The triplet containing the target sample point cloud is used as the target triplet.

[0011] In another achievable implementation, the determining the state of the target object at the current moment based on the pixel values ​​of the pixels in the target area includes: Get the pixel mean and minimum radius for each color; Determine a pixel value range according to the pixel mean and the minimum radius; Determine the color of each pixel point according to the pixel value of each pixel point in the target area and the pixel value range; Obtain the target number of all pixels of the same color in the target area; Based on the number of targets, the color of the target area is determined as the state of the target object at the current moment.

[0012] In another achievable implementation, determining the color of the target area based on the target quantity includes: Based on the target number, a candidate color is obtained, the number of pixel points of the candidate color being greater than the number of pixel points of other colors; Determine a connected area according to the positions of all pixels of the candidate color; If the color proportion of the candidate color in the connected area is greater than a set proportion threshold, the candidate color is determined as the color of the target area.

[0013] In yet another feasible implementation, the target object is a traffic light arranged on the set route.

[0014] In a second aspect, an embodiment of the present invention provides a state recognition device for a target object, comprising: A data acquisition module, used to acquire environmental data collected when the vehicle is traveling along a set route at the current moment, wherein the environmental data at least includes point cloud data and image frames of the target object at the current moment; A benchmark set determination module, used to determine a target triplet matching the point cloud data in a pre-constructed benchmark set, wherein the benchmark set includes at least one triplet, each triplet including a sample image of a target object acquired when the vehicle travels along the set route at a historical moment, a sample point cloud, and a position of the target object in the sample image; A position determination module, used for obtaining the position of the target object in the sample image of the target triplet; A region determination module, configured to determine a target region in the image frame based on the position; The state determination module is used to determine the state of the target object at a current moment based on the pixel values ​​of the pixel points in the target area.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method as described above.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, any of the methods described above is implemented.

[0017] The target object state recognition method, device, equipment and computer storage medium of the embodiments of the present invention use a benchmark set constructed by using data collected when the vehicle was driving at historical moments to find the target area in the image frame where the target object encountered by the vehicle at the current moment is located, and then based on the target area, the current state of the target object can be identified. Under the premise of ensuring accurate recognition of the target object state, the cost of early data collection is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a method for determining the state of a target object provided in an embodiment of the present application; Figure 2 A schematic diagram of a target area provided in an embodiment of the present application; Figure 3 A flowchart of triple construction provided in the embodiment of the present application; Figure 4 A schematic diagram of the original image comparison provided in the embodiment of the present application; Figure 5 A comparison diagram of the area occupied by the target object provided in the embodiment of the present application; Figure 6 A flow chart of a method for identifying the color of a traffic light provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a target state recognition device provided in an embodiment of the present application; Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention, rather than to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0021] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0022] In the field of rail transit, various objects may appear in front of a vehicle during its driving process, which may interfere with or affect the vehicle's current driving state (for example, when an obstacle appears in front, the vehicle may need to enter a deceleration state). For example, taking the target object as a traffic light, the accurate detection and state recognition of the traffic light is a key link to ensure the safe operation of the vehicle.

[0023] At present, the recognition of traffic lights usually relies on deep learning technology, which is achieved through image target detection and classification. These methods often rely on a large amount of manually collected data, which not only has limited interpretability, but is also easily affected by changes in ambient lighting and is difficult to meet strict functional safety standards. There are also methods that use electronic maps to locate traffic lights, which requires the establishment of accurate high-precision maps, manual annotation of the specific locations of traffic lights, and relies on accurate real-time positioning technology.

[0024] In view of the complexity of the current situation, this solution proposes a target state recognition solution. For example, this solution can be applied to identify the state of traffic lights on the road and determine the color displayed by the traffic lights, so that vehicles can plan their driving strategies in advance. Specifically, this solution adopts multi-sensor fusion technology, combined with big data model construction, including at least one benchmark set, which includes several triples, to achieve automatic labeling of traffic lights and abandon the dependence on electronic maps. And the final state of the traffic light is determined by using the pixel value of the pixel point in the target area. This technology not only improves the accuracy and reliability of traffic light recognition, but also greatly reduces the need for manual intervention, thereby providing a more efficient and economical solution for the safe operation of rail transit.

[0025] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0026] Figure 1 This is a flow chart of the target state method provided in the embodiment of the present application. For example, since traffic lights have a significant impact on the safety of vehicle driving, this article focuses on using traffic lights as the target object for detailed description. In practical applications, the target object can be other fixed targets on a fixed route in addition to traffic lights. Figure 1 As shown, the method may include the following steps: Step S110: Acquire environmental data collected when the vehicle is traveling along a set route at the current moment.

[0027] The environmental data at least includes point cloud data and image frames of the target object at the current moment.

[0028] In this embodiment, taking the vehicle as a motor vehicle as an example, the set route can be a marking line on the road (specifically divided into solid lines, dotted lines, double yellow lines, etc.), and the motor vehicle needs to drive on the road according to the set route. Among them, a target object (i.e., a traffic light) may appear during driving. At this time, the motor vehicle needs to determine the corresponding driving strategy (such as deceleration, parking, acceleration, etc.) based on the state of the traffic light (specifically, the state includes red, green or yellow).

[0029] In addition, the vehicle may also refer to a rail train. The train usually has a fixed route during its travel, and at some intersections, traffic lights are set up to inform the train whether it is currently suitable for passage.

[0030] The following uses a train as an example to illustrate how to obtain environmental data.

[0031] In this embodiment, during the running of the train, environmental data can be collected by setting corresponding sensors in front of the train. Exemplarily, the sensor may include a camera, a radar, etc. The camera may capture an image in front of the train. Exemplarily, if a traffic light appears in front of the train, the image captured at this time may be used as an image frame at the current moment. Similarly, the data collected by the radar at this time may be used as point cloud data at the current moment.

[0032] Among them, whether there is a traffic light in front of the train can be determined by performing target detection on the image captured by the camera, or by other observation means (such as manual observation, etc.).

[0033] In addition, in order to reduce the number of image frames and improve the quality of the captured image frames, a distance can be set. For example, when a traffic light appears ahead, the image captured by the camera can be used as an image frame only when the distance between the train and the traffic light is only X meters. X can be set according to actual conditions, for example, the value of X is 150.

[0034] In addition, it should be noted that since data from multiple sensors is involved, in order to ensure the accuracy and consistency of the data, each sensor can be turned on at the same time, and there is no time difference between them. In this way, at the same time, image frames can be obtained from the camera, and point cloud data can also be obtained from the radar. That is, they are all data collected at the same time and share a timestamp.

[0035] Step S120: Determine a target triplet that matches the point cloud data in a pre-constructed benchmark set.

[0036] The benchmark set includes multiple triplets, each of which includes a sample image of the target object, a sample point cloud, and the position of the target object in the sample image when the vehicle is traveling along the set route at a historical moment. These triplets can be sorted in the benchmark set according to the timestamp.

[0037] In this embodiment, the construction of the benchmark set mainly relies on the environmental data collected by the sensors on the vehicle when the vehicle travels along the set route at historical moments.

[0038] Specifically, when the vehicle is traveling at a historical moment, the camera can be used to collect image data in front of the vehicle along the entire line (i.e., the entire set route), and the radar can be used to collect point cloud data in front of the vehicle along the entire line.

[0039] Among them, when a traffic light appears in front of the vehicle, a corresponding traffic light will also appear in the sample image. However, for the vehicle, it is necessary to use a large visual model to detect the collected full-line image data and determine the confidence of each frame of the image, and then select images with high confidence (these images contain traffic lights). In this way, the vehicle can recognize whether there are traffic lights in the sample image.

[0040] In addition, the location of the traffic light in the sample image can be identified by means of target detection models. At the same time, the timestamp of the camera when the sample image was captured can be obtained, and then the corresponding sample point cloud can be found from the full line point cloud data based on the timestamp. In this way, all three elements of the triplet have been obtained.

[0041] In this embodiment, the sample point cloud in the target triplet is matched with the point cloud data at the current moment.

[0042] Step S130: obtaining the position of the target object in the sample image of the target triplet.

[0043] Step S140: Determine a target area in the image frame based on the position.

[0044] In this embodiment, the position of the target object in the sample image mentioned above has been detected and identified by the target detection model. If the point cloud data at the current moment matches the sample point cloud, since the sample point cloud and the sample image share the same timestamp, the image frame at the current moment actually matches the sample image, that is, the position of the target object in the sample image can be directly used to determine the position of the target object in the image frame at the current moment.

[0045] For example, Figure 2 A schematic diagram of the target area provided in the embodiment of the present application, such as Figure 2 As shown, based on the position of the traffic light in the sample image, the target area of ​​the traffic light in the image frame at the current moment is directly determined, which is equivalent to reducing the target detection process and realizing automatic labeling of the target area where the traffic light is located, and the labeling efficiency is faster.

[0046] Step S150: Determine the state of the target object at the current moment based on the pixel values ​​of the pixels in the target area.

[0047] In this embodiment, after the target area is determined in the image frame, the target area can be cut out from the image frame, and then the pixel values ​​of the pixels in the target area are analyzed to determine the color displayed by the traffic light (i.e., the state of the target object).

[0048] Specifically, traffic lights usually have three colors: red, green and yellow. The pixel values ​​of different colors are different. By analyzing the pixel values, the color displayed by the traffic light at the current moment can be determined.

[0049] The traffic light is located in front of the vehicle (e.g., within 150 meters as mentioned above), and the vehicle can determine the driving strategy based on the color of the traffic light. For example, when the traffic light is red, the vehicle can slow down in advance.

[0050] The embodiment of the present application can accurately find the location area of ​​the traffic light in the current image frame by acquiring the current image frame during the vehicle operation phase, and then use the data collected when the vehicle was driving at historical moments, and then perform color recognition on the location area to determine the color state of the traffic light. This method significantly reduces manual input because the preliminary data is mainly collected when the vehicle was driving at historical moments and does not require manual labeling. Moreover, since multiple types of data (including point clouds and two-dimensional images) are used to determine the location area of ​​the traffic light in the current image frame, the accuracy and reliability are relatively higher, thereby improving the accuracy of target state recognition.

[0051] The following describes in detail how to construct the benchmark set mentioned above through some embodiments.

[0052] In this embodiment, it is mentioned above that the vehicle mainly relies on sensors to obtain data (including point clouds and two-dimensional images). Specifically, the camera, lidar, millimeter-wave radar and inertial measurement unit can be integrated into a sensor module to ensure that the relative positions between them remain unchanged to ensure the accuracy of data collection.

[0053] In order to capture the traffic lights in front of the vehicle, the sensor module can be installed at the head of the vehicle so that it faces forward.

[0054] In this embodiment, we first introduce how to construct a triple. After completing the construction of a triple, other triples can be constructed by referring to this method. Figure 3 The triplet construction flow chart provided in the embodiment of the present application continues to take the traffic light as the target object as an example, such as Figure 3 As shown, the specific steps are as follows: Step S310: Acquire sensor data acquired when the vehicle travels along the set route at a historical moment, wherein the sensor data includes at least one original point cloud and at least one original image.

[0055] In this embodiment, for the accuracy of data collection, the vehicle speed can be controlled as much as possible at a set speed at historical moments, and the set speed can be lower than the usual average speed. In this way, when the lighting conditions are good, the vehicle can be driven at a low speed to collect data along the entire line, which can improve the accuracy of the sensor data.

[0056] The sensor data may be obtained by fusing data collected by multiple types of sensors. For example, in some embodiments, the original image may be collected by a camera provided on the vehicle, and the original point cloud may be obtained by an inertial measurement sensor of a radar sensor provided on the vehicle. For example, the radar sensor may include a laser radar and a millimeter wave radar.

[0057] Among them, the original image can be directly obtained by shooting with a camera, and the original point cloud can be motion compensated to the image shooting time through the millimeter wave radar and inertial measurement unit to ensure the time synchronization between the original point cloud and the original image.

[0058] Step S320: Select an image containing a target object from at least one original image as a sample image.

[0059] In this embodiment, in order to avoid missing the traffic lights that appear on the route, the above sensors are always on, and the data collection of the entire line will result in a large amount of data, so some screening is required. Specifically, the collected image data of the entire line can be detected by the visual model first, and the images with higher confidence rankings or greater than the confidence threshold requirement can be selected as sample images, which can reduce the workload of subsequent processing.

[0060] In addition, in some implementations, the acquired original images may be preliminarily screened to screen out some images with defects during camera shooting, and then images containing the target object may be selected from the remaining images after the screening. Specifically, images that do not meet the set conditions may be screened out from all the acquired original images, and then images containing the target object may be selected from the remaining images after the screening as sample images.

[0061] The set requirement may refer to that a size requirement is not met and / or a distance between adjacent images does not meet a specified threshold.

[0062] Specifically, those that do not meet the size requirements can be divided into two categories: Category 1: The area size of the effective information contained in the original image does not meet the requirements. For example, when the vehicle is driving, the camera may be affected by the external environment, causing the camera to be partially blocked, which causes the area size of the effective information contained in it to not meet the requirements. For example, Figure 4The original image comparison diagram provided in the embodiment of the present application is as follows: Figure 4 As shown, there is a partial shadow area in the first original image. This is because the camera head is blocked by obstacles (such as leaves), which results in only a part of the valid information in the entire captured image (the shadow area does not contain any valid information due to lens occlusion).

[0063] Category 2: The size of the area occupied by the target object in the original image does not meet the requirements. For example, Figure 5 The comparison diagram of the area occupied by the target provided in the embodiment of the present application shows that as the vehicle is constantly moving forward, the target in front will become larger and larger in the field of view of the camera. From the perspective of the image, the area occupied by the target in the image is getting larger and larger. Figure 5 As shown, the third original image may be taken when the distance between the vehicle and the traffic light is 2*N meters, and the size of the traffic light in the third original image is still relatively small. The fourth original image may be taken when the distance between the vehicle and the traffic light is only N meters, and the size of the traffic light in the fourth original image is relatively larger, where N is a positive integer. In this way, the size requirement can be set to filter out some original images in which the traffic lights are displayed too small.

[0064] Combined with the above Figure 5 In the actual situation, for the second type of size requirement, the set size can be configured in the following way. Specifically, we can control the distance between the vehicle and the target object to be a set distance threshold (for example, 150 meters), and then obtain the original image captured by the camera on the vehicle at this set distance threshold, and determine the area occupied by the target object in the original image, and then use the area as the set size.

[0065] For example, assuming that during vehicle driving, only the status of traffic lights within 150 meters needs to be identified, then based on the actual size of the traffic lights, the pixel size range of the traffic lights imaged on the original image can be calculated to ensure that there is no interference from targets that are too small.

[0066] In addition, in this embodiment, adjacent original images may refer to: an original image taken at two adjacent moments while the vehicle is driving, thereby obtaining two adjacent original images. If the vehicle speed is too fast, there will be a large distance between the two times. This distance is regarded as the distance between adjacent original images. If this distance is less than a specified threshold, it needs to be screened out.

[0067] The embodiment of the present application configures the set size so that some original images can be screened out, and the size occupied by the targets in these original images is usually larger, which avoids the target being too small and being interfered by other targets, thereby improving the subsequent recognition accuracy of the target.

[0068] Step S330: Obtain the location area of ​​the target object in the sample image.

[0069] In this embodiment, the location area of ​​the traffic light in the sample image can be obtained by using a target detection algorithm. In addition, the location area of ​​the traffic light in the sample image can also be marked by manual marking.

[0070] The traffic light is usually circular, so the location area may refer to a circular area that matches the shape of the traffic light. In addition, in other embodiments, the location area may be set larger, for example, a square to include other areas except the area where the circular traffic light is located.

[0071] Step S340: According to the timestamp of the sample image, a target point cloud collected at the same time as the sample image is obtained in at least one original point cloud as a sample point cloud.

[0072] In this embodiment, the camera records the timestamp of each frame when capturing images. In addition, the radar also records the timestamp when collecting point cloud data. In this way, the point cloud corresponding to the timestamp can be found in the full line point cloud data according to the timestamp of the sample image. This point cloud is the target point cloud, which is subsequently used as the sample point cloud and together with the sample image, constitutes two elements in the triplet.

[0073] Step S350: constructing a triplet based on the sample image, the location area of ​​the target object in the sample image and the sample point cloud.

[0074] In this embodiment, it has been mentioned above that the triplet can be regarded as a data set containing (image, point cloud, traffic light coordinates), the image in the triplet refers to the sample image, the point cloud in the triplet refers to the sample point cloud, and the traffic light coordinates in the triplet refer to the position of the target object in the sample image.

[0075] The embodiment of the present application realizes automatic labeling of traffic light locations by processing the original point clouds and original images collected by multiple sensors, and can quickly and conveniently construct a benchmark set without requiring a large amount of manual intervention to collect training data for the model or manual labeling, thereby reducing the labor cost invested in the early construction of the benchmark set.

[0076] How to determine the target triplet will be described in detail below through some embodiments.

[0077] In this embodiment, the reference set includes at least one triple (usually, the vehicle will collect data along the entire route when driving on the set route, and in fact, there are many triples). Each triple has a corresponding sample point cloud. Among them, by comparing the difference between each sample point cloud and the point cloud data at the current moment, it can be determined whether the sample point cloud matches the point cloud data at the current moment. If they match, the triple in which the sample point cloud is located is regarded as the target triple.

[0078] The matching process may specifically include the following steps: First, a batch of candidate sample point clouds are preliminarily screened out; then, refined screening is performed by comparing feature points to determine the sample point clouds that match the point cloud data at the current moment.

[0079] Specifically, the determination of the target triplet can be divided into two stages, one is the preliminary screening stage, and the other is the high-precision screening stage. The preliminary screening stage mainly selects some candidate sample point clouds that may match the point cloud data at the current moment from a large number of sample point clouds, and the high-precision screening stage mainly matches the candidate sample point clouds more accurately. For example, the first feature point can be selected from the candidate sample point cloud, and the second feature point can also be selected from the point cloud data. By comparing the matching degree between the two feature points, it is determined whether the candidate sample point cloud matches the point cloud data.

[0080] Among them, the voxel matching algorithm can be used for screening in the preliminary screening stage. Specifically, the point cloud data can be divided into M small cubes, the sample point cloud can be divided into N small cubes, and then the overlap between these cubes is calculated. If the overlap between the cube of the point cloud data and the cube of the sample point cloud is high and greater than the overlap threshold, then the point cloud data and the sample point cloud are matched.

[0081] Among them, in the high-precision screening stage, the feature point can refer to one or more, and specifically, some representative feature points can be extracted from the point cloud, such as corner points, edge points, etc. Then, the matching relationship between the two point clouds can be determined by comparing the distance and direction between these feature points.

[0082] In addition, the candidate sample point cloud can also be matched and compared with all points in the point cloud data. This will result in higher matching accuracy. Exemplary methods include the following two methods: (1) Iterative Closest Point (ICP) algorithm iteratively finds the closest point pair between two point clouds, and then updates the position and posture of the point cloud based on the distance and direction between these point pairs. This process is repeated until a certain convergence condition is met.

[0083] (2) The Normal Distributions Transform (NDT) algorithm represents the point cloud as a probability distribution model, and then achieves point cloud matching by minimizing the KL divergence (Kullback-Leibler divergence) between the two models.

[0084] In other embodiments, when the number of triplets contained in the benchmark set is small or the computing resources are abundant, high-performance equipment can be selected to directly compare all feature points in each sample point cloud with all feature points in the point cloud data at the current moment one by one to see if they match, without dividing it into a preliminary screening stage and a high-precision screening stage, which can reduce the complexity of the process.

[0085] The embodiment of the present application implements a point cloud matching strategy by setting a preliminary screening stage and a high-precision screening stage, which can optimize computing performance and reduce the massive data faced during point cloud matching. At the same time, the two screening stages can jointly ensure the accuracy of the final point cloud matching.

[0086] How to determine the state of a target object is described in detail below through some embodiments.

[0087] In this embodiment, the target object is a traffic light set on a set route. For example, the traffic light can display different colors to represent that it is in different states. Among them, a certain color can be described by the color mean and the minimum radius, that is, each color has its corresponding color mean and minimum radius. Among them, the color mean can be obtained in the following way: based on the traffic light open source data set, a fixed threshold method is used to extract the color values ​​of each traffic light in different color states, and add it to the corresponding color set. For example, the color value of a traffic light when the light is red is H1, and the color value of another traffic light when the light is red is H2. They are all added to the red set. Then, the three color sets of red, green and yellow are statistically analyzed to obtain the average value as the color mean.

[0088] For example, if we know that the color mean of red is (x, y, z), then there is a distance between the pixel value of the current pixel and the color mean. If the distance is less than the minimum radius D, it is determined that the pixel belongs to red.

[0089] The setting of the minimum radius needs to be implemented through the following steps: there is a distance between each color value in the color set and the color mean. When setting the minimum radius, it is necessary to satisfy the requirement that a set percentage (for example, 99%) of the color values ​​have a distance from the color mean that is smaller than the minimum radius. The smaller the radius, the better it can reduce misjudgment.

[0090] Furthermore, the color of the target area can be determined by determining the number of pixels belonging to each color for each pixel, so as to serve as the state of the traffic light.

[0091] In some embodiments, the pixels in the target area are classified into red, green, yellow and other four categories, and the number of pixels in the corresponding categories is counted. The color with the highest frequency of occurrence among red, green and yellow is selected as the candidate color. If the appearance ratio is lower than the specified threshold, it is considered that the current traffic light is abnormal. For the candidate color, a connected component analysis can be further performed to determine the proportion of pixels of this color in the connected component area. If the proportion is higher than the set proportion threshold, this color is regarded as the color state of the traffic light.

[0092] In this embodiment, in image processing, connected component analysis is a method for detecting and marking interconnected pixel areas in an image. The connectivity refers to the connection relationship between pixel points. For example, taking a pixel point as an example, one method considers that the four pixel points above, below, left, and right are all connected to it, and there is another method that considers that in addition to the four pixel points above, below, left, and right being connected to it, it is also connected to the four pixel points on its diagonal, that is, there are eight connected pixel points.

[0093] The embodiment of the present application uses an open source data set of traffic lights to obtain the color mean, and then uses a combination of rules and statistical analysis to analyze the color of the target area. Without relying on a large amount of field data, the color displayed by the traffic light can be directly determined to ensure recognition accuracy.

[0094] For example, Figure 6 The flow chart of the traffic light color recognition method provided in the embodiment of the present application is as follows: Figure 6 As shown, it includes the following steps: Step S610: Collect track environment data by camera and lidar simultaneously to identify traffic lights.

[0095] Step S620: Detect the position of the traffic light in the two-dimensional image using the model, and save the position of the traffic light.

[0096] Step S630: Match the current frame point cloud with the previously saved frame to find the point cloud frame that successfully matches the current frame. After the match is successful, use the previous traffic light position to obtain the detection box in the image frame corresponding to the current point cloud frame.

[0097] Step S640: After obtaining the detection frame, perform statistical analysis based on the traffic light open source data set to determine the color range of red, green and yellow, and then perform color analysis on the traffic light detection area of ​​the current frame image to determine the final state.

[0098] In this embodiment, the vehicle is first driven along the entire set route to record the point cloud and image, and determine the position of the traffic light in the image to form a triplet. Then, when the subsequent vehicle is in the motion stage, the new point cloud-image frame is found from the triplet, and the position of the traffic light in the image corresponding to that frame of point cloud is directly taken as the target detection result of the current image frame, as the target area.

[0099] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0100] Figure 7 A schematic diagram of the structure of a target state recognition device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the state identification device 700 includes a data acquisition module 710 , a triplet determination module 720 , a position determination module 730 , an area determination module 740 and a state determination module 750 .

[0101] The data acquisition module 710 is used to acquire the environmental data collected when the vehicle is traveling along the set route at the current moment, wherein the environmental data at least includes the point cloud data and image frames of the target object at the current moment.

[0102] The triplet determination module 720 is used to determine a target triplet that matches the point cloud data in a pre-constructed benchmark set, wherein the benchmark set includes at least one triplet, each of which includes a sample image of a target object, a sample point cloud, and a position of the target object in the sample image when the vehicle travels along a set route at a historical moment; The position determination module 730 is used to obtain the position of the target object in the sample image of the target triplet.

[0103] The region determination module 740 is used to determine a target region in the image frame based on the position.

[0104] The state determination module 750 is used to determine the state of the target object at the current moment based on the pixel values ​​of the pixels in the target area.

[0105] In one example, a triplet construction module is also included, which is used to obtain sensor data obtained when the vehicle is traveling along a set route at a historical moment, and the sensor data includes at least one original point cloud and at least one original image; then, an image containing a target object is selected from the at least one original image as a sample image; and the location area where the target object is located in the sample image is obtained; then, according to the timestamp of the sample image, a target point cloud collected at the same time as the sample image is obtained in the at least one original point cloud as a sample point cloud; finally, a triplet is constructed based on the sample image, the location area where the target object is located in the sample image, and the sample point cloud.

[0106] In one example, the triplet construction module is specifically used to: obtain an original image through a camera provided on the vehicle; and obtain an original point cloud through a radar sensor and an inertial measurement sensor provided on the vehicle.

[0107] In one example, the triplet construction module can be specifically used to: remove images that do not meet set conditions from all acquired original images to obtain remaining images; and then select images containing target objects from the remaining images as sample images.

[0108] The set condition includes at least one of not satisfying a set size and not satisfying a specified threshold value at a distance between adjacent original images.

[0109] In one example, a size determination module is also included, which is used to obtain an image captured when the vehicle is at a set distance threshold from the target object; determine the area occupied by the target object in the image; and determine a set size based on the area.

[0110] In one example, the triplet determination module can be specifically used to: divide the sample point cloud into N cubes; divide the point cloud data into M cubes; when the overlap degree between N cubes and M cubes is greater than the overlap threshold, use the sample point cloud as a candidate sample point cloud, where N and M are both positive integers; obtain the first feature point in each candidate sample point cloud and the second feature point in the point cloud data; when the feature matching degree between the first feature point and the second feature point is greater than the matching threshold, use the candidate sample point cloud as the target sample point cloud; use the triplet containing the target sample point cloud as the target triplet.

[0111] In one example, the state determination module can be specifically used to: obtain the pixel mean and minimum radius of each color; determine the pixel value range based on the pixel mean and minimum radius; determine the color of each pixel point based on the pixel value and pixel value range of each pixel point in the target area; obtain the target number of all pixels of the same color in the target area; based on the target number, determine the color of the target area as the state of the target object at the current moment.

[0112] In one example, the state determination module can be specifically used to: obtain a candidate color based on the target number, the number of pixels of the candidate color is greater than the number of pixels of other colors; then determine the connected area based on the positions of all pixels of the candidate color; wherein, if the color proportion of the candidate color in the connected area is greater than a set proportion threshold, the candidate color is determined as the target area color.

[0113] In one example, the target object is a traffic light located on a set route.

[0114] The device provided in the embodiment of the present application can be used to execute the method in the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0115] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the data acquisition module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above data acquisition module. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0116] Figure 8 The hardware structure diagram of the electronic device provided in the embodiment of the present application is as follows: The electronic device 800 may include a processor 801 and a memory 802 storing computer program instructions.

[0117] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0118] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 802 may include a removable or non-removable (or fixed) medium, or the memory 802 is a non-volatile solid-state memory. The memory 802 may be inside or outside the integrated gateway disaster recovery device.

[0119] In one example, the memory 802 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0120] The memory 802 may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0121] The processor 801 implements the method in the above embodiment by reading and executing the computer program instructions stored in the memory 802 .

[0122] In one example, the electronic device may further include a communication interface 803 and a bus 804. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 803 are connected via a bus 804 and communicate with each other.

[0123] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiment of the present invention.

[0124] Bus 804 includes hardware, software or both, and couples the components of online data traffic billing equipment to each other. For example, but not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industry Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, Memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or two or more of these combinations. Where appropriate, bus 804 may include one or more buses. Although the embodiments of the present invention describe and illustrate specific buses, the present invention contemplates any suitable bus or interconnection.

[0125] In addition, in combination with the method in the above embodiment, the embodiment of the present invention can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the methods in the above embodiment is implemented.

[0126] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the methods in the above embodiments when the computer program is processed and executed.

[0127] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0128] The functional blocks shown in the above structural block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memories (ROM), flash memory, erasable read-only memories (EROM), floppy disks, compact discs (CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0129] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0130] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0131] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A method for identifying the state of a target object, characterized in that: include: Acquire environmental data collected when the vehicle is traveling along a set route at the current moment, wherein the environmental data at least includes point cloud data and image frames of the target object at the current moment; Determining a target triplet matching the point cloud data in a pre-constructed benchmark set, wherein the benchmark set includes at least one triplet, each triplet including a sample image of a target object, a sample point cloud, and a position of the target object in the sample image acquired when the vehicle travels along the set route at a historical moment; Acquire the position of the target object in the sample image of the target triplet; Based on the position, determining a target area in the image frame; The state of the target object at a current moment is determined based on the pixel values ​​of the pixel points in the target area.

2. The method according to claim 1, characterized in that Also includes: Acquire sensor data acquired when the vehicle travels along the set route at a historical moment, the sensor data comprising at least one original point cloud and at least one original image; Selecting an image containing the target object from at least one original image as the sample image; Acquire the location area of ​​the target object in the sample image; According to the timestamp of the sample image, acquiring, in at least one original point cloud, a target point cloud collected at the same time as the sample image as the sample point cloud; The triplet is constructed based on the sample image, the location area where the target object is located in the sample image, and the sample point cloud.

3. The method according to claim 2, characterized in that The obtaining of sensor data acquired when the vehicle travels along the set route at a historical moment includes: Acquiring the original image through a camera provided on the vehicle; The original point cloud is obtained by using a radar sensor and an inertial measurement sensor provided on the vehicle.

4. The method according to claim 2, characterized in that: The selecting an image containing the target object from at least one original image as the sample image comprises: Removing images that do not meet a set condition from all the acquired original images to obtain remaining images, wherein the set condition includes at least one of not meeting a set size and a distance between adjacent original images not meeting a specified threshold; An image containing the target object is selected from the remaining images as the sample image.

5. The method according to claim 4, characterized in that Also includes: Acquire an image captured when the vehicle is at a set distance threshold from the target object; Determine the area occupied by the target object in the image; The set size is determined based on the area of ​​the region.

6. The method according to claim 1, characterized in that The step of determining a target triplet matching the point cloud data in a pre-built benchmark set includes: Divide the sample point cloud into N cubes; Dividing the point cloud data into M cubes; When the degree of overlap between the N cubes and the M cubes is greater than an overlap threshold, the sample point cloud is used as a candidate sample point cloud, where N and M are both positive integers; Obtaining a first feature point in each candidate sample point cloud and a second feature point in the point cloud data; When the feature matching degree between the first feature point and the second feature point is greater than a matching degree threshold, taking the candidate sample point cloud as a target sample point cloud; The triplet containing the target sample point cloud is used as the target triplet.

7. The method according to claim 1, characterized in that The determining the state of the target object at the current moment based on the pixel values ​​of the pixels in the target area includes: Get the pixel mean and minimum radius for each color; Determine a pixel value range according to the pixel mean and the minimum radius; Determine the color of each pixel point according to the pixel value of each pixel point in the target area and the pixel value range; Obtain the target number of all pixels of the same color in the target area; Based on the number of targets, the color of the target area is determined as the state of the target object at the current moment.

8. The method according to claim 7, characterized in that The step of determining the color of the target area based on the target quantity includes: Based on the target number, a candidate color is obtained, the number of pixel points of the candidate color being greater than the number of pixel points of other colors; Determine a connected area according to the positions of all pixels of the candidate color; If the color proportion of the candidate color in the connected area is greater than a set proportion threshold, the candidate color is determined as the color of the target area.

9. The method according to any one of claims 1 to 8, characterized in that The target object is a traffic light arranged on the set route.

10. A device for identifying the state of a target object, characterized in that: include: A data acquisition module, used to acquire environmental data collected when the vehicle is traveling along a set route at the current moment, wherein the environmental data at least includes point cloud data and image frames of the target object at the current moment; A triplet determination module is used to determine a target triplet matching the point cloud data in a pre-constructed benchmark set, wherein the benchmark set includes at least one triplet, each triplet including a sample image of a target object, a sample point cloud, and a position of the target object in the sample image acquired when the vehicle travels along the set route at a historical moment; A position determination module, used for obtaining the position of the target object in the sample image of the target triplet; A region determination module, configured to determine a target region in the image frame based on the position; The state determination module is used to determine the state of the target object at a current moment based on the pixel values ​​of the pixel points in the target area.

11. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer storage medium stores computer program instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images