End-to-end signal light recognition method and apparatus, and device, medium and vehicle

Abstract

The present disclosure relates to an end-to-end signal light recognition method and apparatus, and a device, a medium and a vehicle. The signal light recognition method comprises: acquiring, from a target vehicle, an image sequence to be subjected to prediction; inputting said image sequence into a pre-trained machine learning model, and performing feature extraction on said image sequence by a residual network module in the machine learning model, so as to obtain a first feature vector in a preset dimension; inputting the first feature vector into an encoding processing module in the machine learning model for encoding processing, so as to obtain a second feature vector; and inputting the second feature vector into a classification module in the machine learning model, and recognizing the state of a signal light by the classification module according to the second feature vector, so as to obtain a signal light recognition result. Therefore, an end-to-end signal light recognition solution can be implemented by using a pre-trained machine learning model, thereby improving the rate of signal light recognition, and reducing the complexity of signal light recognition and the maintenance cost.

Description

End-to-end signal light recognition method, device, equipment, medium and vehicle

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on June 16, 2023, with application number 202310722552.2 and application name “End-to-end traffic light recognition method, device, equipment, medium and vehicle”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present disclosure relates to the field of autonomous driving technology, and is related to, but not limited to, an end-to-end signal light recognition method, device, equipment, medium, and vehicle. Background Art

[0004] With the increasing prevalence of intelligent vehicles, more and more vehicles are equipped with Navigate on Autopilot (NOA) capabilities. This feature is further divided into highway NOA and urban NOA. Urban NOA requires autonomous vehicles to be able to navigate structured urban roads. Therefore, it is necessary to recognize traffic lights and then make decisions and control the vehicle based on the light status (such as red or green).

[0005] In existing traffic light recognition, it is necessary to first obtain a specific traffic light image (i.e., an image containing only traffic lights) from images collected by the vehicle (such as environmental images or driving images), then analyze the traffic light image (such as analyzing bulb color and bulb classification), and finally determine the traffic light status based on the analysis results. This method is extremely complex to implement, and the processes of obtaining traffic light images from vehicle images and identifying traffic light status cannot be combined into a holistic solution, that is, an end-to-end recognition solution cannot be implemented. In addition, due to the wide variety of traffic light styles, even relying on a large amount of manual annotation to obtain massive traffic light images as training data is difficult to solve the long-tail problem.

[0006] Therefore, how to avoid the long-tail problem and directly achieve end-to-end traffic light recognition using images collected by vehicles without obtaining specific traffic light images has become an urgent problem to be solved.

[0007] Summary of the Invention

[0008] In order to solve the above technical problems, the present disclosure provides an end-to-end traffic light recognition method, device, equipment, medium and vehicle, which can use a pre-trained machine learning model to implement an end-to-end traffic light recognition solution, thereby improving the rate of traffic light recognition and reducing the complexity and maintenance cost of traffic light recognition.

[0009] A first aspect of the present disclosure provides an image data annotation method, the method comprising:

[0010] Obtain image sequences and vehicle trajectory information when the vehicle passes through the intersection;

[0011] Determine the vehicle's driving intention and driving process information based on trajectory information;

[0012] Annotation information is generated according to the driving intention and driving process information of the vehicle, and the image sequence is annotated according to the annotation information; the annotation information is the status of the signal light.

[0013] A second aspect of the present disclosure provides a model training method, the method comprising:

[0014] Acquire a training sample set, the training sample set comprising a plurality of annotated image sequences, each annotated image sequence comprising at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotating method described in the first aspect above;

[0015] The training sample set is used to train the machine learning model to be trained to obtain a pre-trained machine learning model.

[0016] A third aspect of the present disclosure provides a signal light recognition method, the method comprising:

[0017] Obtain the image sequence to be predicted of the target vehicle;

[0018] Inputting the image sequence to be predicted into a pre-trained machine learning model, and performing feature extraction on the image sequence to be predicted by the residual network module in the machine learning model to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in the second aspect above;

[0019] Inputting the first feature vector into an encoding processing module in a machine learning model for encoding processing to obtain a second feature vector;

[0020] The second eigenvector is input into a classification module in the machine learning model, and the classification module identifies the state of the traffic light according to the second eigenvector to obtain a traffic light recognition result.

[0021] A fourth aspect of the embodiments of the present disclosure provides a vehicle start-stop control method, the method comprising:

[0022] Obtaining the target driving intention of the target vehicle and the position information of the preset stop line;

[0023] When the traffic light recognition result corresponding to the target driving intention is a red light, controlling the target vehicle to stop at a position corresponding to a preset stop line, wherein the traffic light recognition result is obtained by the traffic light recognition method described in the third aspect above;

[0024] When the signal light recognition result corresponding to the target driving intention is green, the target vehicle is controlled to continue driving at the position corresponding to the preset stop line.

[0025] A fifth aspect of the present disclosure provides a data tagging device, the device comprising:

[0026] An image sequence acquisition module configured to acquire an image sequence and vehicle trajectory information when a vehicle passes through an intersection;

[0027] The image sequence annotation module is configured to determine the vehicle's driving intention and driving process information based on the trajectory information; generate annotation information based on the vehicle's driving intention and driving process information, and annotate the image sequence based on the annotation information; the annotation information is the status of the traffic light.

[0028] A sixth aspect of the present disclosure provides a model training device, the device comprising:

[0029] a sample set acquisition module configured to acquire a training sample set, the training sample set including a plurality of annotated image sequences, each annotated image sequence including at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotation method described in the first aspect;

[0030] The model training module is configured to train the machine learning model to be trained using the training sample set to obtain a pre-trained machine learning model.

[0031] A seventh aspect of the embodiments of the present disclosure provides a signal light recognition device, the device comprising:

[0032] A data acquisition module configured to acquire a sequence of images to be predicted of a target vehicle;

[0033] a feature extraction module configured to input the image sequence to be predicted into a pre-trained machine learning model, and to extract features from the image sequence to be predicted by a residual network module in the machine learning model to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in the second aspect above;

[0034] an encoding processing module configured to input the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector;

[0035] The traffic light recognition module is configured to input the second eigenvector into the classification module in the machine learning model, and the classification module recognizes the state of the traffic light according to the second eigenvector to obtain a traffic light recognition result.

[0036] An eighth aspect of the present disclosure provides a vehicle control device, the device comprising:

[0037] An information acquisition module configured to acquire target driving intentions of a target vehicle and position information of a preset stop line;

[0038] a first control module configured to control the target vehicle to stop at a position corresponding to a preset stop line when a signal light recognition result corresponding to the target driving intention is red, wherein the signal light recognition result is obtained by the signal light recognition method described in the third aspect;

[0039] The second control module is configured to control the target vehicle to continue traveling at a position corresponding to a preset stop line when a signal light recognition result corresponding to the target driving intention is green.

[0040] A ninth aspect of the present disclosure provides an electronic device, the device including:

[0041] Memory;

[0042] processor; and

[0043] A computer program, wherein the computer program is stored in a memory and is configured to be executed by a processor to implement the image data labeling method of the first aspect, the model training method of the second aspect, the signal light recognition method of the third aspect, and the vehicle start-stop control method of the fourth aspect.

[0044] The tenth aspect of the embodiments of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image data labeling method of the first aspect, the model training method of the second aspect, the signal light recognition method of the third aspect, and the vehicle start-stop control method of the fourth aspect are implemented.

[0045] An eleventh aspect of an embodiment of the present disclosure provides a vehicle comprising the electronic device of the ninth aspect described above.

[0046] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0047] The end-to-end traffic light recognition method, apparatus, device, medium and vehicle provided by the embodiments of the present disclosure can obtain a sequence of images to be predicted of a target vehicle. After obtaining the sequence of images to be predicted, the sequence of images to be predicted is input into a pre-trained machine learning model. The residual network module in the machine learning model performs feature extraction on the sequence of images to be predicted to obtain a first feature vector of a preset dimension. The first feature vector is input into an encoding processing module in the machine learning model for encoding processing to obtain a second feature vector. The second feature vector is then input into a classification module in the machine learning model. The classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result. The feature vector corresponding to the sequence of images to be predicted can be extracted by the pre-trained machine learning model to realize traffic light recognition. That is, no additional manual intervention is required. The pre-trained machine learning model is used to realize an end-to-end traffic light recognition solution, which improves the rate of traffic light recognition and reduces the complexity and maintenance cost of traffic light recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0049] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] FIG1 is a flow chart of an image data annotation method provided by an embodiment of the present disclosure;

[0051] FIG2 is a flow chart of a signal light recognition method provided by an embodiment of the present disclosure;

[0052] FIG3 is a schematic diagram of the structure of a data tagging device provided by an embodiment of the present disclosure;

[0053] FIG4 is a schematic structural diagram of a model training device provided by an embodiment of the present disclosure;

[0054] FIG5 is a schematic structural diagram of a signal light recognition device and a vehicle control device provided by an embodiment of the present disclosure;

[0055] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0058] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. All other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0059] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0060] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0061] Normally, in existing signal light recognition, it is necessary to first obtain a specific signal light image (i.e., an image containing only signal lights) from an image collected by a vehicle (such as an environmental image or a driving image), and then analyze the signal light image, such as analyzing the color of the bulb, the classification of the bulb, etc., and finally determine the signal light status based on the analysis results; the implementation process of this technical solution is extremely complicated, and the processes of obtaining the signal light image from the image collected by the vehicle and identifying the signal light status cannot be done in an overall solution, that is, an end-to-end recognition solution cannot be implemented; in addition, due to the variety of signal light styles, even if a large amount of manual annotation is relied upon to obtain a large amount of signal light images as training data, it is difficult to solve the long-tail problem. Therefore, how to avoid the long-tail problem and directly implement end-to-end signal light recognition using images collected by the vehicle without obtaining specific signal light images has become an urgent problem to be solved. In response to this problem, the embodiments of the present disclosure provide a signal light recognition method, which is described below in conjunction with specific embodiments.

[0062] In an embodiment of the present disclosure, a traffic light recognition method is to identify a target vehicle's image sequence to be predicted based on a pre-trained machine learning model to obtain a traffic light recognition result, wherein the pre-trained machine learning model is obtained by annotating image data to obtain annotated image data, and then the annotated image data is used as a sample training set to train the machine learning model to be trained. Therefore, the automatic annotation process of image data is first introduced in detail.

[0063] FIG1 is a flowchart of an image data annotation method provided by an embodiment of the present disclosure. In the embodiment of the present disclosure, the method for annotating image data may specifically include steps S110 - S130 .

[0064] S110 : Acquire an image sequence and vehicle trajectory information when the vehicle passes through an intersection.

[0065] In the embodiment of the present disclosure, the image sequence is composed of N images, at least one of the N images includes an image of a traffic light, where N is a positive integer, for example, N=10.

[0066] Optionally, the signal light can be a red light, a traffic light, or other signal lights used to direct traffic operations, etc. The specific signal light can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0067] In the embodiment of the present disclosure, the vehicle's trajectory information may include the vehicle's driving status information, wherein the vehicle's driving status information may specifically include the vehicle's driving speed information, driving acceleration information, Global Positioning System (GPS) positioning information and other information related to the vehicle's driving process. The specific content of the vehicle's driving status information is not limited in the embodiment of this application.

[0068] S120: Determine the vehicle's driving intention and driving process information based on the trajectory information.

[0069] In the embodiment of the present disclosure, the driving intention of the vehicle includes any one of turning left, going straight, turning right, and making a U-turn, which is used to indicate the driving intention of the vehicle when passing through an intersection.

[0070] Specifically, the driving process information includes two situations: passing and stopping, which are used to indicate the driving status of the vehicle when passing through the intersection. Among them, when the driving process information is passing, it indicates that the vehicle did not stop when passing through the intersection, so it can correspond to a green light. When the driving process information is stopping, it indicates that the vehicle stopped when passing through the intersection, so it can correspond to a red light. More specifically, the vehicle's driving intention can be obtained using trajectory information. For example, the positional relationship between trajectory points in the trajectory can be used to determine whether the vehicle is turning left, going straight, turning right, or turning around, and the corresponding behavior can be used as the vehicle's driving intention. Similarly, the positional relationship between trajectory points in the trajectory can also be used to determine whether the vehicle is stopping. Since the determination of the vehicle's driving intention and driving process information based on trajectory information can be achieved using existing methods, it will not be repeated here.

[0071] S130 : Generate annotation information according to the driving intention and driving process information of the vehicle, and annotate the image sequence according to the annotation information.

[0072] In an embodiment of the present disclosure, labeling information is generated based on the vehicle's driving intention and driving process information, and the vehicle's driving intention and driving process information can be combined to generate the labeling information; specifically, the labeling information is the status of the traffic light, and the labeling information may include left turn red light, left turn green light, right turn red light, right turn green light, straight ahead red light, straight ahead green light, U-turn red light, U-turn green light; the labeling information can be used to correspond to the status of the traffic light at the current intersection.

[0073] In an embodiment of the present disclosure, after annotating an image sequence according to annotation information, an annotated image sequence can be obtained, and then the annotated image sequence can be used as a training sample, thus obtaining a training sample. Each annotated image sequence can be used as a training sample, that is, by annotating at least one image sequence, at least one annotated image sequence can be obtained, thus obtaining at least one training sample.

[0074] Based on the above embodiments of the present disclosure, a process of training a machine learning model to be trained to obtain a pre-trained machine learning model is specifically introduced.

[0075] The process of training the machine learning model to be trained to obtain a pre-trained machine learning model can specifically include: obtaining a training sample set, using the training sample set to train the machine learning model to be trained, and obtaining a pre-trained machine learning model.

[0076] In the disclosed embodiment, obtaining a training sample set may specifically include: obtaining at least one training sample using the aforementioned method, and using the set corresponding to the at least one training sample as the training sample set. In the disclosed embodiment, training a machine learning model to be trained using the training sample set may specifically include: training the machine learning model to be trained using a supervised learning approach based on a sequence of labeled images in the training sample set, thereby obtaining a pre-trained machine learning model.

[0077] In the embodiments of the present disclosure, the process of training the machine learning model to be trained can be implemented using existing methods, which are not limited here.

[0078] In an embodiment of the present disclosure, after obtaining the training sample set, the method may further include: performing standardization processing on the image data in the training sample set, and training the machine learning model to be trained based on the standardized training sample set.

[0079] It should be noted that the standardization process may include image cropping, image size adjustment, image format adjustment and other processes, so as to obtain a training sample set of a preset size and format.

[0080] Exemplarily, the preset size is an image of 960*512 pixels.

[0081] In the embodiment of the present disclosure, the training sample set can be standardized before training the machine learning model to be trained, thereby improving the training rate.

[0082] Furthermore, the corresponding traffic lights during vehicle driving are identified based on the pre-trained machine learning model.

[0083] Figure 2 is a flowchart of a signal light recognition method provided by an embodiment of the present disclosure. The method can be executed by a signal light recognition device, which can be implemented in software and / or hardware. The signal light recognition device can be configured in an electronic device, such as a server or a terminal or a server cluster, wherein the terminal can specifically include a vehicle-mounted terminal, etc., or can be any device that can be used to process the signal light recognition method, etc.

[0084] As shown in FIG2 , in an embodiment of the present disclosure, a signal light recognition method is provided, which may specifically include the following steps:

[0085] S210: Obtain a sequence of images to be predicted of the target vehicle.

[0086] In an embodiment of the present disclosure, the electronic device can obtain a sequence of images to be predicted of the target vehicle in response to the target vehicle being in a driving state.

[0087] The image sequence to be predicted can be understood as data collected during the target vehicle's driving process and used to identify the status of traffic lights. It can include image data of the target vehicle at intersections as well as image data of the target vehicle at non-intersections. The image sequence to be predicted also consists of N images, where N is a positive integer, for example, N=10.

[0088] Optionally, the image sequence to be predicted can be N images collected in real time by the target vehicle during the driving process of the target vehicle, or can be N images obtained from the video collected in real time by the target vehicle based on a preset method as the image sequence to be predicted. The preset method can be an existing method such as continuous extraction, frame extraction, etc., which is not limited here.

[0089] S220: Input the image sequence to be predicted into a pre-trained machine learning model, and use the residual network module in the machine learning model to extract features of the image sequence to be predicted to obtain a first feature vector of a preset dimension.

[0090] In an embodiment of the present disclosure, after obtaining the image sequence to be predicted of the target vehicle, the electronic device inputs the image sequence to be predicted into a pre-trained machine learning model, and the residual network module in the machine learning model performs feature extraction on the image sequence to be predicted to obtain a first feature vector of a preset dimension.

[0091] Optionally, the preset dimension may be 1024 dimensions, or other dimension values. The specific preset dimension may be determined based on actual conditions, and the embodiments of the present application do not limit this.

[0092] S230. Input the first eigenvector into the encoding processing module in the machine learning model for encoding processing to obtain a second eigenvector.

[0093] In an embodiment of the present disclosure, after obtaining the first eigenvector, the electronic device inputs the first eigenvector into an encoding processing module in a machine learning model for encoding processing to obtain a second eigenvector.

[0094] Optionally, the encoding processing module can be an encoder in a transformer module.

[0095] S240. Input the second eigenvector into a classification module in the machine learning model, and the classification module identifies the state of the traffic light according to the second eigenvector to obtain a traffic light recognition result.

[0096] In an embodiment of the present disclosure, after obtaining the second eigenvector, the electronic device inputs the second eigenvector into a classification module in a machine learning model, and the classification module identifies the state of the traffic light according to the second eigenvector to obtain a traffic light recognition result.

[0097] Optionally, the classification module can be composed of a binary classifier based on the softmax function.

[0098] The classification module can include four binary classifiers, each corresponding to a driving intent, such as a left turn binary classifier, a right turn binary classifier, a straight ahead binary classifier, and a U-turn binary classifier. Based on the classification module, the state of traffic lights at an intersection can be identified, and the corresponding traffic light recognition results for each driving intent can be obtained.

[0099] Specifically, after obtaining the second eigenvector, the electronic device inputs the second eigenvector into the classification module in the machine learning model. The classification module determines the eigenvalue corresponding to the second eigenvector, and obtains the traffic light recognition result based on the eigenvalue corresponding to the second eigenvector and the one-to-one correspondence between the eigenvalue and the traffic light state.

[0100] The one-to-one correspondence between the characteristic value and the traffic light status is pre-acquired and stored data.

[0101] For example, if the signal light state corresponding to the characteristic value within the first preset range is red, the recognition result is red; if the signal light state corresponding to the characteristic value within the second preset range is green, the recognition result is green.

[0102] In the case that there are multiple traffic lights corresponding to multiple driving intentions in the image sequence to be predicted, the obtained traffic light recognition result is the recognition result of multiple traffic lights.

[0103] For example, in the image sequence to be predicted, there are three traffic lights at a certain intersection, namely a left turn light, a straight ahead light, and a right turn light. Then the obtained traffic light recognition results may be a left turn green light, a straight ahead red light, and a right turn green light.

[0104] In the disclosed embodiment, a sequence of images to be predicted of a target vehicle can be obtained. After obtaining the sequence of images to be predicted, the sequence of images to be predicted is input into a pre-trained machine learning model. The residual network module in the machine learning model performs feature extraction on the sequence of images to be predicted to obtain a first feature vector of a preset dimension. The first feature vector is input into an encoding processing module in the machine learning model for encoding processing to obtain a second feature vector. The second feature vector is then input into a classification module in the machine learning model. The classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result. The feature vector corresponding to the sequence of images to be predicted can be extracted by the pre-trained machine learning model to realize traffic light recognition. That is, no additional manual intervention is required. An end-to-end traffic light recognition solution is realized by using the pre-trained machine learning model, which improves the rate of traffic light recognition while reducing the complexity and maintenance cost of traffic light recognition.

[0105] In some embodiments of the present disclosure, obtaining the image sequence to be predicted of the target vehicle in S210 may specifically include: obtaining a driving image video of the target vehicle based on an image acquisition device installed on the target vehicle; and performing frame extraction processing on the driving image video at a preset frequency to obtain the image sequence to be predicted.

[0106] In the embodiment of the present disclosure, the image acquisition device may be a camera, a webcam, or other device for acquiring images, and the number of the image acquisition device may be at least one.

[0107] Optionally, the preset frequency may be an interval of a preset distance, or an interval of a preset time, etc.

[0108] Specifically, the electronic device can start the image acquisition device in response to the target vehicle being started or the target vehicle being in a driving state, and the image acquisition device can capture the driving image of the target vehicle in real time. After the target vehicle's driving image video is captured, the driving image video is frame-processed at a preset frequency to obtain a sequence of images to be predicted.

[0109] Exemplarily, if the preset frequency is 5 meters and the preset distance is 5 meters, the electronic device can extract a corresponding image from the driving image video every 5 meters the target vehicle travels, and then extract N images, and use the N images as the image sequence to be predicted; if the preset frequency is 1 second and the preset time is 1 second, the electronic device can extract a corresponding image from the driving image video every 1 second, and then extract N images to obtain the image sequence to be predicted; the electronic device can also extract all images corresponding to the target vehicle traveling 5 meters from the driving image video, and extract a corresponding image every 1 second from all images corresponding to the target vehicle traveling 5 meters, and then extract N images to obtain the image sequence to be predicted.

[0110] In the embodiment of the present disclosure, the driving image video of the target vehicle collected in real time by the image acquisition device can be frame-extracted at a preset frequency to obtain a sequence of images to be predicted, and then the signal light status can be identified based on the sequence of images to be predicted, thereby improving the efficiency of signal light recognition.

[0111] In an embodiment of the present disclosure, after obtaining the signal light recognition result, the vehicle start-stop control method can be continued to be executed. The vehicle start-stop control method may include: obtaining the target driving intention of the target vehicle and the position information of the preset stop line; when the signal light recognition result corresponding to the target driving intention is a red light, controlling the target vehicle to stop driving at the position corresponding to the preset stop line; when the signal light recognition result corresponding to the target driving intention is a green light, controlling the target vehicle to continue driving at the position corresponding to the preset stop line.

[0112] In the embodiment of the present disclosure, the position information of the preset stop line is determined based on a preset stop line recognition model and internal and external parameters of the image acquisition device.

[0113] Specifically, after acquiring the image sequence to be identified, the electronic device inputs the image sequence to be identified into a pre-trained stop line recognition model, and the pre-trained stop line recognition model recognizes the image sequence to be identified and outputs a stop line recognition result, wherein the stop line recognition result includes the position information of the stop line in the image. The electronic device calculates the position information of the preset stop line based on the position information of the stop line in the image and the internal and external parameters of the image acquisition device.

[0114] Among them, the target driving intention can be obtained using the navigation route, such as using the relationship between the current position of the target vehicle and the next or next few position points in the navigation planning trajectory to determine the target driving intention of the vehicle; the method of obtaining the target driving intention of the vehicle and the position information of the preset stop line can also be achieved using other existing methods, which are not limited here.

[0115] For example, when the target driving intention of the target vehicle is to turn left and the signal light recognition result is a red light for turning left, the target vehicle is controlled to stop driving at the position corresponding to the preset stop line; when the target driving intention of the target vehicle is to turn right and the signal light recognition result is a green light for turning right, the target vehicle is controlled to continue driving at the position corresponding to the preset stop line.

[0116] In the embodiment of the present disclosure, the target driving intention of the target vehicle and the position information of the preset stop line can be obtained. According to the signal light recognition result and the target driving intention of the target vehicle, the target vehicle is controlled to stop or continue driving at the preset stop line position. It has been applied in urban NOA and improved the accuracy of signal light recognition.

[0117] Figure 3 is a schematic diagram of the structure of a data annotation device provided by an embodiment of the present disclosure. The data annotation device in the embodiment of the present disclosure can be set in an electronic device, which can be a server, a terminal, or a server cluster. The terminal can specifically include an in-vehicle terminal, etc., which is not limited here.

[0118] As shown in FIG3 , the data annotation apparatus 300 may include an image sequence acquisition module 310 and an image sequence annotation module 320 .

[0119] The image sequence acquisition module 310 may be configured to acquire an image sequence of a vehicle passing through an intersection and the trajectory information of the vehicle.

[0120] The image sequence annotation module 320 can be configured to determine the vehicle's driving intention and driving process information based on the trajectory information; generate annotation information based on the vehicle's driving intention and driving process information, and annotate the image sequence based on the annotation information; the annotation information is the status of the traffic light.

[0121] In the embodiment of the present disclosure, by acquiring an image sequence of a vehicle passing through an intersection and the trajectory information of the vehicle, the driving intention and driving process information of the vehicle can be determined based on the trajectory information; annotation information can be generated based on the driving intention and driving process information of the vehicle, and the image sequence can be annotated based on the annotation information, thereby realizing automatic annotation of the data and improving the accuracy of data annotation.

[0122] In some embodiments of the present disclosure, the trajectory information of the vehicle includes the driving state information of the vehicle; the driving intention of the vehicle includes any one of turning left, going straight, turning right, and making a U-turn.

[0123] In some embodiments of the present disclosure, the image sequence consists of N images, wherein at least one of the N images includes an image of a traffic light.

[0124] It should be noted that the data annotation device 300 shown in FIG3 can execute each step in the above-mentioned image data annotation method embodiment and realize each process and effect in the above-mentioned image data annotation method embodiment, which will not be described in detail here.

[0125] Figure 4 is a schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure. The model training device in the embodiment of the present disclosure can be set in an electronic device, which can be a server, a terminal, or a server cluster. The terminal can specifically include an in-vehicle terminal, etc., which is not limited here.

[0126] As shown in FIG. 4 , the model training device 400 may include a sample set acquisition module 410 and a model training module 420 .

[0127] The sample set acquisition module 410 may be configured to acquire a training sample set, where the training sample set includes at least one annotated image sequence, wherein the annotated image sequence is obtained by the image data annotation method described in the above embodiment.

[0128] The model training module 420 can be configured to train the machine learning model to be trained using the training sample set to obtain a pre-trained machine learning model.

[0129] In some embodiments of the present disclosure, the model training module 420 may also be configured to perform normalization processing on the image data in the training sample set after obtaining the training sample set, and train the machine learning model to be trained based on the normalized training sample set.

[0130] In some embodiments of the present disclosure, the model training module 420 can be specifically configured to train the machine learning model to be trained based on the labeled image sequence in the training sample set using a supervised learning method to obtain a pre-trained machine learning model.

[0131] It should be noted that the model training device 400 shown in Figure 4 can execute the various steps in the above-mentioned model training method embodiment and realize the various processes and effects in the above-mentioned model training method embodiment, which will not be described in detail here.

[0132] Figure 5 is a schematic diagram of the structure of a signal light recognition device and a vehicle control device provided in an embodiment of the present disclosure. The signal light recognition device and the vehicle control device in the embodiment of the present disclosure can be installed in an electronic device, which can be a server, a terminal, or a server cluster. The terminal can specifically include an in-vehicle terminal, etc., which is not limited here.

[0133] As shown in FIG. 5 , the signal light recognition device 500 may include a data acquisition module 510 , a feature extraction module 520 , a coding processing module 530 and a signal light recognition module 540 .

[0134] The data acquisition module 510 may be configured to acquire a sequence of images to be predicted of the target vehicle.

[0135] The feature extraction module 520 can be configured to input the image sequence to be predicted into a pre-trained machine learning model, and the residual network module in the machine learning model performs feature extraction on the image sequence to be predicted to obtain a first feature vector of a preset dimension. The pre-trained machine learning model is obtained by the model training method described in the above method embodiment.

[0136] The encoding processing module 530 can be configured to input the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector.

[0137] The traffic light recognition module 540 may be configured to input the second feature vector into a classification module in a machine learning model, and the classification module may recognize the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

[0138] In the disclosed embodiment, a sequence of images to be predicted of a target vehicle can be obtained. After obtaining the sequence of images to be predicted, the sequence of images to be predicted is input into a pre-trained machine learning model. The residual network module in the machine learning model performs feature extraction on the sequence of images to be predicted to obtain a first feature vector of a preset dimension. The first feature vector is input into an encoding processing module in the machine learning model for encoding processing to obtain a second feature vector. The second feature vector is then input into a classification module in the machine learning model. The classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result. The feature vector corresponding to the sequence of images to be predicted can be extracted by the pre-trained machine learning model to realize traffic light recognition. That is, no additional manual intervention is required. An end-to-end traffic light recognition solution is realized by using the pre-trained machine learning model, which improves the rate of traffic light recognition while reducing the complexity and maintenance cost of traffic light recognition.

[0139] In some embodiments of the present disclosure, the data acquisition module 510 can be specifically configured to obtain the driving image video of the target vehicle based on the image acquisition device installed on the target vehicle; and perform frame extraction processing on the driving image video at a preset frequency to obtain the image sequence to be predicted.

[0140] In some embodiments of the present disclosure, the image sequence to be predicted is N images collected by the target vehicle during driving and used to identify the state of the traffic light.

[0141] As shown in FIG. 5 , the vehicle control device 600 may include an information acquisition module 610 , a first control module 620 , and a second control module 630 .

[0142] The information acquisition module 610 may be configured to acquire the target driving intention of the target vehicle and the position information of a preset stop line.

[0143] The first control module 620 can be configured to control the target vehicle to stop driving at a position corresponding to a preset stop line when the signal light recognition result corresponding to the target driving intention is a red light, wherein the signal light recognition result is obtained by any signal light recognition method described in any one of the above method embodiments.

[0144] The second control module 630 may be configured to control the target vehicle to continue traveling at a position corresponding to a preset stop line when the traffic light recognition result corresponding to the target driving intention is green.

[0145] In the embodiment of the present disclosure, the target driving intention of the target vehicle and the position information of the preset stop line can be obtained. According to the signal light recognition result and the target driving intention of the target vehicle, the target vehicle is controlled to stop or continue driving at the preset stop line position. It has been applied in urban NOA and improved the accuracy of signal light recognition.

[0146] In some embodiments of the present disclosure, the position information of the preset stop line is determined based on a preset stop line recognition model and internal and external parameters of the image acquisition device.

[0147] It should be noted that the signal light recognition device 500 shown in Figure 5 can execute the various steps in the above-mentioned signal light recognition method embodiment, and realize the various processes and effects in the above-mentioned signal light recognition method embodiment, which will not be elaborated here; at the same time, the vehicle control device 600 shown in Figure 5 can execute the various steps in the above-mentioned vehicle start-stop control method embodiment, and realize the various processes and effects in the above-mentioned vehicle start-stop control method embodiment, which will not be elaborated here.

[0148] FIG6 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0149] In the embodiment of the present disclosure, the electronic device shown in FIG6 may be a server, a terminal, or a server cluster, wherein the terminal may specifically include a vehicle-mounted terminal, etc., which is not limited here.

[0150] As shown in FIG. 6 , the electronic device may include a processor 660 and a memory 662 storing computer program instructions.

[0151] Specifically, the processor 660 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0152] Memory 662 may include a large-capacity memory for information or instructions. By way of example, and not limitation, memory 662 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 662 may include removable or non-removable (or fixed) media. Where appropriate, memory 662 may be internal or external to the integrated gateway device. In certain embodiments, memory 662 is non-volatile solid-state memory. In certain embodiments, memory 662 includes read-only memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, programmable ROM (PROM), electrical programmable ROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0153] The processor 660 reads and executes the computer program instructions stored in the memory 662 to perform the steps of the image data labeling method, model training method, signal light recognition method and vehicle start-stop control method provided in the embodiments of the present disclosure.

[0154] In one example, the electronic device may further include a transceiver 664 and a bus 666. As shown in FIG6, the processor 660, the memory 662, and the transceiver 664 are connected via the bus 666 and communicate with each other.

[0155] The bus 666 may include hardware, software, or both. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of the above. Where appropriate, the bus 666 may include one or more buses.

[0156] The embodiments of the present disclosure also provide a computer-readable storage medium, which can store a computer program. When the computer program is executed by a processor, the processor implements the image data labeling method, model training method, signal light recognition method and vehicle start-stop control method provided by the embodiments of the present disclosure.

[0157] The above-mentioned storage medium may, for example, include a memory 662 of computer program instructions, and the above-mentioned instructions may be executed by the processor 660 of the electronic device to complete the image data annotation method, model training method, signal light recognition method, and vehicle start-stop control method provided in the embodiments of the present disclosure. Alternatively, the storage medium may be a non-transitory computer-readable storage medium, for example, a non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] The embodiments of the present disclosure further provide a vehicle, which includes the above-mentioned electronic device and can implement the various processes and effects in the above-mentioned embodiments of the present disclosure, which will not be described in detail here.

[0159] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or apparatus.

[0160] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein. Industrial Applicability

[0161] The embodiments of the present application provide an end-to-end signal light recognition method, apparatus, device, medium, and vehicle. The signal light recognition method includes: obtaining a sequence of images to be predicted of a target vehicle; inputting the sequence of images to be predicted into a pre-trained machine learning model, and having the residual network module in the machine learning model perform feature extraction on the sequence of images to be predicted to obtain a first feature vector of a preset dimension; inputting the first feature vector into an encoding processing module in the machine learning model for encoding processing to obtain a second feature vector; inputting the second feature vector into a classification module in the machine learning model, and having the classification module recognize the state of the signal light according to the second feature vector to obtain a signal light recognition result. By implementing the above method, an end-to-end signal light recognition solution can be implemented using a pre-trained machine learning model, thereby improving the rate of signal light recognition and reducing the complexity and maintenance cost of signal light recognition.

Claims

1. An image data annotation method, the method comprising: Acquire an image sequence of a vehicle passing through an intersection and trajectory information of the vehicle; Determining the driving intention and driving process information of the vehicle according to the trajectory information; Annotation information is generated according to the driving intention and driving process information of the vehicle, and the image sequence is annotated according to the annotation information; the annotation information is the state of the traffic light.

2. The image data annotation method according to claim 1, wherein: The trajectory information of the vehicle includes the driving state information of the vehicle; the driving intention of the vehicle includes any one of turning left, going straight, turning right, and making a U-turn.

3. The image data annotation method according to claim 1, wherein: The image sequence consists of N images, wherein at least one image among the N images includes an image of a traffic light, and N is a positive integer.

4. A model training method, the method comprising: Acquire a training sample set, wherein the training sample set includes a plurality of annotated image sequences, each of which includes at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotation method according to any one of claims 1 to 3; The training sample set is used to train the machine learning model to be trained to obtain a pre-trained machine learning model.

5. The model training method according to claim 4, wherein: After obtaining the training sample set, the method further includes: The image data in the training sample set is standardized, and the machine learning model to be trained is trained based on the standardized training sample set.

6. The model training method according to claim 4, wherein: The step of training the machine learning model to be trained by using the training sample set to obtain the pre-trained machine learning model includes: Based on the labeled image sequence in the training sample set, the machine learning model to be trained is trained using a supervised learning method to obtain the pre-trained machine learning model.

7. A signal light recognition method, the method comprising: Obtain a sequence of images to be predicted of the target vehicle; The image sequence to be predicted is input into a pre-trained machine learning model, and a residual network module in the machine learning model performs feature extraction on the image sequence to be predicted to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in any one of claims 4 to 6; Inputting the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector; The second feature vector is input into a classification module in the machine learning model, and the classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

8. The signal light recognition method according to claim 7, wherein: The step of obtaining a sequence of images to be predicted of the target vehicle includes: Based on the image acquisition device installed on the target vehicle, obtaining the driving image video of the target vehicle; The driving image video is subjected to frame extraction processing at a preset frequency to obtain the image sequence to be predicted.

9. The signal light recognition method according to claim 7, wherein: The image sequence to be predicted is N images collected by the target vehicle during driving and used to identify the state of the traffic light, where N is a positive integer.

10. A vehicle start-stop control method, the method comprising: Obtain target driving intention of target vehicle and location information of preset stop line; When the signal light recognition result corresponding to the target driving intention is a red light, controlling the target vehicle to stop driving at the position corresponding to the preset stop line, wherein the signal light recognition result is obtained by the signal light recognition method described in any one of claims 7 to 9; When the signal light recognition result corresponding to the target driving intention is a green light, the target vehicle is controlled to continue driving at the position corresponding to the preset stop line.

11. The vehicle start-stop control method according to claim 10, wherein: The position information of the preset stop line is determined based on a preset stop line recognition model and internal and external parameters of the image acquisition device.

12. A data annotation device, comprising: An image sequence acquisition module, configured to acquire an image sequence of a vehicle passing through an intersection and trajectory information of the vehicle; The image sequence annotation module is configured to determine the driving intention and driving process information of the vehicle according to the trajectory information; generate annotation information according to the driving intention and driving process information of the vehicle, and annotate the image sequence according to the annotation information; the annotation information is the state of the traffic light.

13. A model training device, comprising: a sample set acquisition module configured to acquire a training sample set, wherein the training sample set includes a plurality of annotated image sequences, each of which includes at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotation method according to any one of claims 1 to 3; The model training module is configured to train the machine learning model to be trained using the training sample set to obtain a pre-trained machine learning model.

14. A signal light recognition device, the device comprising: A data acquisition module, configured to acquire a sequence of images to be predicted of a target vehicle; A feature extraction module, configured to input the image sequence to be predicted into a pre-trained machine learning model, and extract features of the image sequence to be predicted by a residual network module in the machine learning model to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in any one of claims 4 to 6; an encoding processing module, configured to input the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector; The traffic light recognition module is configured to input the second feature vector into the classification module in the machine learning model, and the classification module recognizes the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

15. A vehicle control device, the device comprising: An information acquisition module configured to acquire target driving intention of a target vehicle and position information of a preset stop line; A first control module, configured to control the target vehicle to stop driving at a position corresponding to the preset stop line when the signal light recognition result corresponding to the target driving intention is a red light, wherein the signal light recognition result is obtained by the signal light recognition method described in any one of claims 7 to 9; The second control module is configured to control the target vehicle to continue driving at the position corresponding to the preset stop line when the signal light recognition result corresponding to the target driving intention is green.

16. An electronic device comprising: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 11.

17. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 11.

18. A vehicle comprising the electronic device according to claim 16.

Images