End-to-end signal lamp identification method, device and equipment, medium and vehicle

Abstract

The invention relates to an end-to-end signal lamp recognition method, device and equipment, a medium and a vehicle, and the signal lamp recognition method comprises the steps: obtaining a to-be-predicted image sequence of a target vehicle; inputting the to-be-predicted image sequence into a pre-trained machine learning model, and performing feature extraction on the to-be-predicted image sequence by a residual network module in the machine learning model to obtain a first feature vector of a preset dimension; inputting the first feature vector into a coding processing module in a machine learning model for coding processing to obtain a second feature vector; and inputting the second feature vector into a classification module in a machine learning model, and identifying the state of the signal lamp by the classification module according to the second feature vector to obtain a signal lamp identification result, so that an end-to-end signal lamp identification scheme can be realized by utilizing the pre-trained machine learning model, the signal lamp identification rate is improved, and the signal lamp identification efficiency is improved. And meanwhile, the complexity and maintenance cost of signal lamp identification are reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to an end-to-end signal light recognition method, device, equipment, medium and vehicle. Background Art

[0002] With the popularization of vehicle intelligence, more and more vehicles have begun to have the Navigate on Autopilot (NOA) function, which is further divided into highway NOA and urban NOA. Urban NOA requires autonomous vehicles to be able to drive on structured roads in cities, so it is necessary to identify traffic lights and then complete subsequent tasks such as decision-making and vehicle control based on the status of traffic lights (such as red or green lights).

[0003] In the existing traffic light recognition, it is necessary to first obtain a specific traffic light image (i.e., an image containing only traffic lights) from an image collected by a vehicle (such as an environmental image or a driving image), then analyze the traffic light image (such as analyzing the color of the bulb, classifying the bulb, etc.), and finally determine the state of the traffic light based on the analysis result. The implementation process of this method is extremely complicated. The process of obtaining the traffic light image from the image collected by the vehicle and identifying the state of the traffic light cannot be achieved in an overall solution, that is, an end-to-end recognition solution cannot be implemented. In addition, due to the variety of traffic light styles, even if a large amount of manual annotation is relied upon to obtain a large amount of traffic light images as training data, it is difficult to solve the long-tail problem.

[0004] Therefore, how to avoid the long-tail problem and directly realize end-to-end traffic light recognition using images collected by vehicles without obtaining specific traffic light images has become an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides an end-to-end signal light recognition method, device, equipment, medium and vehicle.

[0006] A first aspect of an embodiment of the present disclosure provides an image data annotation method, the method comprising:

[0007] Obtain image sequences and vehicle trajectory information when the vehicle passes through the intersection;

[0008] Determine the vehicle's driving intention and driving process information based on the trajectory information;

[0009] Annotation information is generated according to the driving intention and driving process information of the vehicle, and the image sequence is annotated according to the annotation information.

[0010] A second aspect of the present disclosure provides a model training method, the method comprising:

[0011] Acquire a training sample set, the training sample set includes a plurality of annotated image sequences, each annotated image sequence includes at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotating method described in the first aspect above;

[0012] The training sample set is used to train the machine learning model to be trained to obtain a pre-trained machine learning model.

[0013] A third aspect of the embodiments of the present disclosure provides a signal light recognition method, the method comprising:

[0014] Obtaining a sequence of images to be predicted of the target vehicle;

[0015] Inputting the image sequence to be predicted into a pre-trained machine learning model, and extracting features of the image sequence to be predicted by a residual network module in the machine learning model to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in the second aspect above;

[0016] Inputting the first feature vector into an encoding processing module in a machine learning model for encoding processing to obtain a second feature vector;

[0017] The second eigenvector is input into a classification module in the machine learning model, and the classification module identifies the state of the traffic light according to the second eigenvector to obtain a traffic light recognition result.

[0018] A fourth aspect of the embodiments of the present disclosure provides a vehicle start-stop control method, the method comprising:

[0019] Obtaining the target driving intention of the target vehicle and the position information of the preset stop line;

[0020] When the signal light recognition result corresponding to the target driving intention is a red light, controlling the target vehicle to stop driving at a position corresponding to a preset stop line, wherein the signal light recognition result is obtained by the signal light recognition method described in the third aspect above;

[0021] When the signal light recognition result corresponding to the target driving intention is green, the target vehicle is controlled to continue driving at the position corresponding to the preset stop line.

[0022] A fifth aspect of the embodiments of the present disclosure provides a data labeling device, the device comprising:

[0023] An image sequence acquisition module is used to acquire an image sequence and vehicle trajectory information when a vehicle passes through an intersection;

[0024] The image sequence annotation module is used to determine the vehicle's driving intention and driving process information based on the trajectory information; generate annotation information based on the vehicle's driving intention and driving process information, and annotate the image sequence based on the annotation information.

[0025] A sixth aspect of the embodiments of the present disclosure provides a model training device, the device comprising:

[0026] A sample set acquisition module, used to acquire a training sample set, the training sample set includes a plurality of annotated image sequences, each annotated image sequence includes at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotation method described in the first aspect above;

[0027] The model training module is used to train the machine learning model to be trained using the training sample set to obtain a pre-trained machine learning model.

[0028] A seventh aspect of an embodiment of the present disclosure provides a signal light recognition device, the device comprising:

[0029] A data acquisition module, used to acquire a sequence of images to be predicted of a target vehicle;

[0030] A feature extraction module, used to input the image sequence to be predicted into a pre-trained machine learning model, and extract features of the image sequence to be predicted by the residual network module in the machine learning model to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in the second aspect above;

[0031] An encoding processing module, used for inputting the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector;

[0032] The traffic light recognition module is used to input the second feature vector into the classification module in the machine learning model, and the classification module recognizes the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

[0033] An eighth aspect of the embodiments of the present disclosure provides a vehicle control device, the device comprising:

[0034] An information acquisition module is used to obtain the target driving intention of the target vehicle and the position information of the preset stop line;

[0035] A first control module, configured to control the target vehicle to stop driving at a position corresponding to a preset stop line when the signal light recognition result corresponding to the target driving intention is a red light, wherein the signal light recognition result is obtained by the signal light recognition method described in the third aspect above;

[0036] The second control module is used to control the target vehicle to continue driving at a position corresponding to a preset stop line when the signal light recognition result corresponding to the target driving intention is a green light.

[0037] A ninth aspect of an embodiment of the present disclosure provides an electronic device, the device comprising:

[0038] Memory;

[0039] Processor; and

[0040] A computer program, wherein the computer program is stored in a memory and is configured to be executed by a processor to implement the image data annotation method of the first aspect, the model training method of the second aspect, the signal light recognition method of the third aspect, and the vehicle start-stop control method of the fourth aspect.

[0041] The tenth aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image data labeling method of the first aspect, the model training method of the second aspect, the signal light recognition method of the third aspect, and the vehicle start-stop control method of the fourth aspect are implemented.

[0042] An eleventh aspect of the embodiments of the present disclosure provides a vehicle, comprising the electronic device of the ninth aspect described above.

[0043] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0044] The end-to-end traffic light recognition method, apparatus, device, medium and vehicle provided by the embodiments of the present disclosure can obtain a sequence of images to be predicted of a target vehicle. After obtaining the sequence of images to be predicted, the sequence of images to be predicted is input into a pre-trained machine learning model. The residual network module in the machine learning model performs feature extraction on the sequence of images to be predicted to obtain a first feature vector of a preset dimension. The first feature vector is input into an encoding processing module in the machine learning model for encoding processing to obtain a second feature vector. The second feature vector is then input into a classification module in the machine learning model. The classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result. The feature vector corresponding to the sequence of images to be predicted can be extracted through the pre-trained machine learning model to realize traffic light recognition. That is to say, no additional manual intervention is required. The pre-trained machine learning model is used to realize an end-to-end traffic light recognition solution, which improves the rate of traffic light recognition and reduces the complexity and maintenance cost of traffic light recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 is a flow chart of an image data annotation method provided by an embodiment of the present disclosure;

[0048] Figure 2 is a flow chart of a signal light recognition method provided by an embodiment of the present disclosure;

[0049] Figure 3 is a structural schematic diagram of a data labeling device provided by an embodiment of the present disclosure;

[0050] Figure 4 is a structural schematic diagram of a model training device provided by an embodiment of the present disclosure;

[0051] Figure 5 It is a structural schematic diagram of a signal light recognition device and a vehicle control device provided by an embodiment of the present disclosure;

[0052] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0055] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0056] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0057] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0058] Typically, in existing signal light recognition, it is necessary to first obtain a specific signal light image (i.e., an image containing only signal lights) from an image captured by a vehicle (such as an environmental image or a driving image), and then analyze the signal light image, such as analyzing the color of the bulb, the classification of the bulb, etc., and finally determine the signal light status based on the analysis results; the implementation process of this technical solution is extremely complicated, and the processes of obtaining the signal light image from the image captured by the vehicle and identifying the signal light status cannot be achieved in an overall solution, that is, an end-to-end recognition solution cannot be achieved; in addition, due to the variety of signal light styles, even if a large amount of manual annotation is relied upon to obtain a large amount of signal light images as training data, it is difficult to solve the long-tail problem. Therefore, how to avoid the long-tail problem and directly realize end-to-end signal light recognition using images captured by the vehicle without obtaining specific signal light images has become an urgent problem to be solved. In response to this problem, the embodiments of the present disclosure provide a signal light recognition method, which is described below in conjunction with specific embodiments.

[0059] In the disclosed embodiment, the traffic light recognition method is to identify the target vehicle's image sequence to be predicted based on a pre-trained machine learning model to obtain a traffic light recognition result, wherein the pre-trained machine learning model is obtained by annotating image data to obtain annotated image data, and then using the annotated image data as a sample training set to train the machine learning model to be trained. Therefore, the automatic annotation process of image data is first introduced in detail.

[0060] Figure 1It is a flowchart of an image data annotation method provided by an embodiment of the present disclosure. In the embodiment of the present disclosure, the method for annotating image data may specifically include steps S110-S130.

[0061] S110, acquiring an image sequence and vehicle trajectory information when the vehicle passes through an intersection.

[0062] In the embodiment of the present disclosure, the image sequence is composed of N images, at least one of the N images includes an image of a traffic light, wherein N is a positive integer, for example, N=10.

[0063] Optionally, the signal light may be a traffic light, a traffic signal light, or other signal lights for directing traffic movement, and the like.

[0064] In the disclosed embodiment, the vehicle's trajectory information may include the vehicle's driving status information, wherein the vehicle's driving status information may specifically include the vehicle's driving speed information, driving acceleration information, GPS positioning information, and other information related to the vehicle's driving process.

[0065] S120: Determine the driving intention and driving process information of the vehicle according to the trajectory information.

[0066] In the disclosed embodiment, the driving intention of the vehicle includes any one of turning left, going straight, turning right, and making a U-turn, which is used to indicate the driving intention of the vehicle when passing through an intersection.

[0067] Specifically, the driving process information includes two situations, namely, passing and stopping, which are used to indicate the driving status of the vehicle when passing through an intersection. Among them, passing indicates that the vehicle does not stop when passing through an intersection, so it can correspond to a green light, and stopping indicates that the vehicle stops when passing through an intersection, so it can correspond to a red light. More specifically, the vehicle's driving intention can be obtained using trajectory information. For example, it can be determined whether the vehicle has any of the following situations: turning left, going straight, turning right, or turning around, through existing methods such as the positional relationship between trajectory points in the trajectory, and the corresponding behavior can be used as the vehicle's driving intention; similarly, it can also be determined whether the vehicle has stopped through existing methods such as the positional relationship between trajectory points in the trajectory. Since determining the vehicle's driving intention and driving process information based on trajectory information can be achieved using existing methods, it will not be repeated here.

[0068] S130 , generating annotation information according to the driving intention and driving process information of the vehicle, and annotating the image sequence according to the annotation information.

[0069] In an embodiment of the present disclosure, labeling information is generated based on the vehicle's driving intention and driving process information, and the vehicle's driving intention and driving process information can be combined to generate the labeling information; specifically, the labeling information may include left turn red light, left turn green light, right turn red light, right turn green light, straight ahead red light, straight ahead green light, U-turn red light, U-turn green light; the labeling information can be used to correspond to the status of the traffic light at the current intersection.

[0070] After annotating the image sequence according to the annotation information, a labeled image sequence can be obtained, and then the labeled image sequence can be used as a training sample, thus obtaining a training sample. Each labeled image sequence can be used as a training sample, that is, by annotating at least one image sequence, at least one labeled image sequence can be obtained, thus obtaining at least one training sample.

[0071] Based on the above embodiments of the present disclosure, a process of training a machine learning model to be trained to obtain a pre-trained machine learning model is specifically introduced.

[0072] The process of training the machine learning model to be trained to obtain a pre-trained machine learning model may specifically include: obtaining a training sample set, and using the training sample set to train the machine learning model to be trained to obtain a pre-trained machine learning model.

[0073] In the embodiment of the present disclosure, obtaining a training sample set may specifically include: obtaining at least one training sample using the aforementioned method, and using the set corresponding to the at least one training sample as the training sample set. In the embodiment of the present disclosure, training a machine learning model to be trained using a training sample set may specifically include: training the machine learning model to be trained using a supervised learning method based on a sequence of labeled images in the training sample set, and obtaining a pre-trained machine learning model.

[0074] In the embodiments of the present disclosure, the process of training the machine learning model to be trained can be implemented using existing methods, which are not limited here.

[0075] In an embodiment of the present disclosure, after acquiring the training sample set, the method may further include: performing standardization processing on the image data in the training sample set, and training the machine learning model to be trained based on the standardized training sample set.

[0076] The standardization process may include image cropping, image size adjustment, image format adjustment and other processes, so as to obtain a training sample set of a preset size and format.

[0077] Exemplarily, the preset size is an image of 960*512 pixels.

[0078] In the embodiments of the present disclosure, the training sample set can be standardized before training the machine learning model to be trained, thereby improving the training rate.

[0079] Furthermore, the corresponding traffic lights during vehicle driving are identified based on the pre-trained machine learning model.

[0080] Figure 2 It is a flow chart of a signal light recognition method provided by an embodiment of the present disclosure. The method can be executed by a signal light recognition device. The signal light recognition device can be implemented in software and / or hardware. The signal light recognition device can be configured in an electronic device, such as a server or a terminal or a server cluster, wherein the terminal can specifically include a vehicle-mounted terminal, etc., or can be any device that can be used to process the signal light recognition method, etc.

[0081] like Figure 2 As shown, in the embodiment of the present disclosure, the signal light recognition method provided may specifically include the following steps:

[0082] S210: Obtain a sequence of images to be predicted of the target vehicle.

[0083] In the embodiment of the present disclosure, the electronic device can acquire a sequence of images to be predicted of the target vehicle in response to the target vehicle being in a driving state.

[0084] The image sequence to be predicted can be understood as data collected during the driving process of the target vehicle for identifying the state of the traffic light, which can include image data of the target vehicle at an intersection, or image data of the target vehicle at a non-intersection. The image sequence to be predicted is also composed of N images, where N is a positive integer, for example, N=10.

[0085] Optionally, the image sequence to be predicted can be N images collected in real time by the target vehicle during the driving process of the target vehicle, or can be N images obtained from the video collected in real time by the target vehicle based on a preset method as the image sequence to be predicted. The preset method can be existing methods such as continuous extraction and frame extraction, which are not limited here.

[0086] S220, inputting the image sequence to be predicted into a pre-trained machine learning model, and using the residual network module in the machine learning model to extract features from the image sequence to be predicted, to obtain a first feature vector of a preset dimension.

[0087] In the disclosed embodiment, after acquiring the image sequence to be predicted of the target vehicle, the electronic device inputs the image sequence to be predicted into a pre-trained machine learning model, and the residual network module in the machine learning model performs feature extraction on the image sequence to be predicted to obtain a first feature vector of a preset dimension.

[0088] Optionally, the preset dimension may be 1024 dimensions.

[0089] S230. Input the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector.

[0090] In the embodiment of the present disclosure, after obtaining the first feature vector, the electronic device inputs the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain the second feature vector.

[0091] Optionally, the encoding processing module may be an encoder in a transformer module.

[0092] S240. Input the second feature vector into a classification module in the machine learning model, and the classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

[0093] In the disclosed embodiment, after obtaining the second eigenvector, the electronic device inputs the second eigenvector into a classification module in the machine learning model, and the classification module identifies the state of the traffic light according to the second eigenvector to obtain a traffic light recognition result.

[0094] Optionally, the classification module may be composed of a binary classifier based on a softmax function.

[0095] The classification module may include four binary classifiers, each of which corresponds to a driving intention, such as a left turn binary classifier, a right turn binary classifier, a straight drive binary classifier, and a U-turn binary classifier. Based on the classification module, the state of the traffic light at an intersection can be identified, and then the identification result of the traffic light corresponding to each driving intention can be obtained.

[0096] Specifically, after obtaining the second eigenvector, the electronic device inputs the second eigenvector into the classification module in the machine learning model, and the classification module determines the eigenvalue corresponding to the second eigenvector, and obtains the traffic light recognition result based on the eigenvalue corresponding to the second eigenvector and the one-to-one correspondence between the eigenvalue and the traffic light state.

[0097] The one-to-one correspondence between the characteristic value and the state of the traffic light is pre-acquired and stored data.

[0098] Exemplarily, the signal light state corresponding to the feature value within the first preset range is red light, and the obtained recognition result is red light; the signal light state corresponding to the feature value within the second preset range is green light, and the obtained recognition result is green light.

[0099] When there are multiple signal lights corresponding to multiple driving intentions in the image sequence to be predicted, the obtained signal light recognition result is the recognition result of multiple signal lights.

[0100] For example, in the image sequence to be predicted, there are three traffic lights at a certain intersection, namely a left turn light, a straight ahead light, and a right turn light. Then the recognition results of the traffic lights may be a left turn green light, a straight ahead red light, and a right turn green light.

[0101] In the disclosed embodiment, a to-be-predicted image sequence of a target vehicle can be acquired. After acquiring the to-be-predicted image sequence, the to-be-predicted image sequence is input into a pre-trained machine learning model. The residual network module in the machine learning model performs feature extraction on the to-be-predicted image sequence to obtain a first feature vector of a preset dimension. The first feature vector is input into an encoding processing module in the machine learning model for encoding processing to obtain a second feature vector. The second feature vector is then input into a classification module in the machine learning model. The classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result. The feature vector corresponding to the to-be-predicted image sequence can be extracted through the pre-trained machine learning model to realize traffic light recognition. That is to say, no additional manual intervention is required. An end-to-end traffic light recognition solution is realized by using the pre-trained machine learning model, which improves the rate of traffic light recognition while reducing the complexity and maintenance cost of traffic light recognition.

[0102] In some embodiments of the present disclosure, obtaining the image sequence to be predicted of the target vehicle in S210 may specifically include: obtaining a driving image video of the target vehicle based on an image acquisition device installed on the target vehicle; performing frame extraction processing on the driving image video at a preset frequency to obtain the image sequence to be predicted.

[0103] In the embodiment of the present disclosure, the image acquisition device may be a camera, a webcam or other device for acquiring images, and the number of the image acquisition device may be at least one.

[0104] Optionally, the preset frequency may be an interval of a preset distance, or an interval of a preset time, etc.

[0105] Specifically, the electronic device can start the image acquisition device in response to the target vehicle being started or the target vehicle being in a driving state, and the image acquisition device can acquire the driving image of the target vehicle in real time. After acquiring the driving image video of the target vehicle, the driving image video is subjected to frame extraction processing at a preset frequency to obtain an image sequence to be predicted.

[0106] Exemplarily, if the preset frequency is 5 meters, the electronic device can extract a corresponding image from the driving image video every 5 meters the target vehicle travels, and then extract N images, and use the N images as the image sequence to be predicted; if the preset frequency is 1 second, the electronic device can extract a corresponding image from the driving image video every 1 second, and then extract N images to obtain the image sequence to be predicted; the electronic device can also extract all images corresponding to the target vehicle traveling 5 meters from the driving image video, and extract a corresponding image every 1 second from all images corresponding to the target vehicle traveling 5 meters, and then extract N images to obtain the image sequence to be predicted.

[0107] In the disclosed embodiment, it is possible to extract frames from the driving image video of the target vehicle captured in real time by an image acquisition device at a preset frequency to obtain a sequence of images to be predicted, and then identify the state of the traffic light based on the sequence of images to be predicted, thereby improving the efficiency of traffic light recognition.

[0108] In an embodiment of the present disclosure, after obtaining the signal light recognition result, the vehicle start-stop control method can continue to be executed, and the vehicle start-stop control method can include: obtaining the target driving intention of the target vehicle and the position information of a preset stop line; when the signal light recognition result corresponding to the target driving intention is a red light, controlling the target vehicle to stop driving at the position corresponding to the preset stop line; when the signal light recognition result corresponding to the target driving intention is a green light, controlling the target vehicle to continue driving at the position corresponding to the preset stop line.

[0109] In the embodiment of the present disclosure, the position information of the preset stop line is determined based on the preset stop line recognition model and the internal and external parameters of the image acquisition device.

[0110] Specifically, after acquiring the image sequence to be identified, the electronic device inputs the image sequence to be identified into a pre-trained stop line recognition model, and the pre-trained stop line recognition model recognizes the image sequence to be identified and outputs a stop line recognition result, wherein the stop line recognition result includes position information of the stop line in the image, and the electronic device calculates the position information of the preset stop line based on the position information of the stop line in the image and the internal and external parameters of the image acquisition device.

[0111] Among them, the target driving intention can be obtained by using the navigation route, such as using the relationship between the current position of the target vehicle and the next or next few position points in the navigation planning trajectory to determine the target driving intention of the vehicle; the method of obtaining the target driving intention of the vehicle and the position information of the preset stop line can also be implemented by other existing methods, which are not limited here.

[0112] Exemplarily, when the target driving intention of the target vehicle is to turn left and the traffic light recognition result is a red light for turning left, the target vehicle is controlled to stop driving at the position corresponding to the preset stop line; when the target driving intention of the target vehicle is to turn right and the traffic light recognition result is a green light for turning right, the target vehicle is controlled to continue driving at the position corresponding to the preset stop line.

[0113] In the disclosed embodiment, the target driving intention of the target vehicle and the position information of the preset stop line can be obtained. According to the signal light recognition result and the target driving intention of the target vehicle, the target vehicle is controlled to stop or continue driving at the preset stop line position. This has been applied in urban NOA and has improved the accuracy of signal light recognition.

[0114] Figure 3 1 is a schematic diagram of the structure of a data labeling device provided in an embodiment of the present disclosure. The data labeling device in the embodiment of the present disclosure can be set in an electronic device, and the electronic device can be a server or a terminal or a server cluster, wherein the terminal can specifically include a vehicle-mounted terminal, etc., which is not limited here.

[0115] like Figure 3 As shown, the data annotation device 300 may include an image sequence acquisition module 310 and an image sequence annotation module 320 .

[0116] The image sequence acquisition module 310 may be used to acquire an image sequence of a vehicle passing through an intersection and the trajectory information of the vehicle.

[0117] The image sequence annotation module 320 can be used to determine the driving intention and driving process information of the vehicle according to the trajectory information; generate annotation information according to the driving intention and driving process information of the vehicle, and annotate the image sequence according to the annotation information.

[0118] In the embodiments of the present disclosure, by acquiring an image sequence of a vehicle passing through an intersection and trajectory information of the vehicle, the driving intention and driving process information of the vehicle can be determined based on the trajectory information; annotation information can be generated based on the driving intention and driving process information of the vehicle, and the image sequence can be annotated based on the annotation information, thereby realizing automatic annotation of data and improving the accuracy of data annotation.

[0119] In some embodiments of the present disclosure, the trajectory information of the vehicle includes the driving state information of the vehicle; the driving intention of the vehicle includes any one of turning left, going straight, turning right, and making a U-turn.

[0120] In some embodiments of the present disclosure, the image sequence consists of N images, wherein at least one image among the N images includes an image of a traffic light.

[0121] It should be noted that Figure 3 The data annotation device 300 shown can execute each step in the above-mentioned image data annotation method embodiment, and realize each process and effect in the above-mentioned image data annotation method embodiment, which will not be described in detail here.

[0122] Figure 4 : is a schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure. The model training device in the embodiment of the present disclosure can be set in an electronic device, and the electronic device can be a server or a terminal or a server cluster, wherein the terminal can specifically include a vehicle-mounted terminal, etc., which is not limited here.

[0123] like Figure 4 As shown, the model training device 400 may include a sample set acquisition module 410 and a model training module 420.

[0124] The sample set acquisition module 410 may be used to acquire a training sample set, where the training sample set includes at least one annotated image sequence, wherein the annotated image sequence is obtained by the image data annotation method described in the above embodiment.

[0125] The model training module 420 can be used to train the machine learning model to be trained using the training sample set to obtain a pre-trained machine learning model.

[0126] In some embodiments of the present disclosure, the model training module 420 may also be used to standardize the image data in the training sample set after acquiring the training sample set, and train the machine learning model to be trained based on the standardized training sample set.

[0127] In some embodiments of the present disclosure, the model training module 420 can be specifically used to train the machine learning model to be trained using a supervised learning method based on a labeled image sequence in a training sample set to obtain a pre-trained machine learning model.

[0128] It should be noted that Figure 4 The model training device 400 shown can execute each step in the above-mentioned model training method embodiment, and realize each process and effect in the above-mentioned model training method embodiment, which will not be described in detail here.

[0129] Figure 5 The structure diagram of a signal light recognition device and a vehicle control device provided in an embodiment of the present disclosure is shown in FIG. The signal light recognition device and the vehicle control device in the embodiment of the present disclosure can be set in an electronic device, and the electronic device can be a server or a terminal or a server cluster, wherein the terminal can specifically include a vehicle-mounted terminal, etc., which is not limited here.

[0130] like Figure 5As shown, the signal light recognition device 500 may include a data acquisition module 510 , a feature extraction module 520 , a coding processing module 530 and a signal light recognition module 540 .

[0131] The data acquisition module 510 may be used to acquire a sequence of images to be predicted of the target vehicle.

[0132] The feature extraction module 520 can be used to input the image sequence to be predicted into a pre-trained machine learning model, and the residual network module in the machine learning model performs feature extraction on the image sequence to be predicted to obtain a first feature vector of a preset dimension. The pre-trained machine learning model is obtained by the model training method described in the above method embodiment.

[0133] The encoding processing module 530 can be used to input the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector.

[0134] The traffic light recognition module 540 can be used to input the second feature vector into the classification module in the machine learning model, and the classification module recognizes the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

[0135] In the disclosed embodiment, a to-be-predicted image sequence of a target vehicle can be acquired. After acquiring the to-be-predicted image sequence, the to-be-predicted image sequence is input into a pre-trained machine learning model. The residual network module in the machine learning model performs feature extraction on the to-be-predicted image sequence to obtain a first feature vector of a preset dimension. The first feature vector is input into an encoding processing module in the machine learning model for encoding processing to obtain a second feature vector. The second feature vector is then input into a classification module in the machine learning model. The classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result. The feature vector corresponding to the to-be-predicted image sequence can be extracted through the pre-trained machine learning model to realize traffic light recognition. That is to say, no additional manual intervention is required. An end-to-end traffic light recognition solution is realized by using the pre-trained machine learning model, which improves the rate of traffic light recognition while reducing the complexity and maintenance cost of traffic light recognition.

[0136] In some embodiments of the present disclosure, the data acquisition module 510 can be specifically used to acquire a driving image video of the target vehicle based on an image acquisition device installed on the target vehicle; and perform frame extraction processing on the driving image video at a preset frequency to obtain an image sequence to be predicted.

[0137] In some embodiments of the present disclosure, the image sequence to be predicted is N images collected by the target vehicle during driving and used to identify the state of the traffic light.

[0138] like Figure 5 As shown, the vehicle control device 600 may include an information acquisition module 610 , a first control module 620 and a second control module 630 .

[0139] The information acquisition module 610 may be used to acquire the target driving intention of the target vehicle and the position information of the preset stop line.

[0140] The first control module 620 can be used to control the target vehicle to stop driving at a position corresponding to a preset stop line when the signal light recognition result corresponding to the target driving intention is a red light, wherein the signal light recognition result is obtained by any signal light recognition method described in any one of the above method embodiments.

[0141] The second control module 630 can be used to control the target vehicle to continue driving at a position corresponding to a preset stop line when the signal light recognition result corresponding to the target driving intention is a green light.

[0142] In the disclosed embodiment, the target driving intention of the target vehicle and the position information of the preset stop line can be obtained. According to the signal light recognition result and the target driving intention of the target vehicle, the target vehicle is controlled to stop or continue driving at the preset stop line position. This has been applied in urban NOA and has improved the accuracy of signal light recognition.

[0143] In some embodiments of the present disclosure, the position information of the preset stop line is determined based on a preset stop line recognition model and internal and external parameters of the image acquisition device.

[0144] It should be noted that Figure 5 The signal light recognition device 500 shown can execute the various steps in the above signal light recognition method embodiment, and realize the various processes and effects in the above signal light recognition method embodiment, which will not be described in detail here; Figure 5 The vehicle control device 600 shown can execute each step in the above-mentioned vehicle start-stop control method embodiment, and realize each process and effect in the above-mentioned vehicle start-stop control method embodiment, which will not be described in detail here.

[0145] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0146] In the disclosed embodiment, Figure 6 The electronic device shown may be a server or a terminal or a server cluster, wherein the terminal may specifically include a vehicle-mounted terminal, etc., which is not limited here.

[0147] like Figure 6 As shown, the electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0148] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0149] The memory 620 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 620 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 620 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 620 is a non-volatile solid-state memory. In a particular embodiment, the memory 620 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0150] The processor 610 reads and executes computer program instructions stored in the memory 620 to perform the steps of the image data labeling method, model training method, signal light recognition method and vehicle start-stop control method provided in the embodiments of the present disclosure.

[0151] In one example, the electronic device may further include a transceiver 630 and a bus 640. Figure 6 As shown, the processor 610, the memory 620 and the transceiver 630 are connected via a bus 640 and communicate with each other.

[0152] The bus 640 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 640 may include one or more buses.

[0153] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the image data labeling method, model training method, signal light recognition method, and vehicle start-stop control method provided in the embodiments of the present disclosure.

[0154] The above-mentioned storage medium may, for example, include a memory 620 of computer program instructions, and the above-mentioned instructions may be executed by the processor 610 of the electronic device to complete the image data annotation method, model training method, signal light recognition method and vehicle start-stop control method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0155] The embodiments of the present disclosure also provide a vehicle, which includes the above-mentioned electronic device and can implement the various processes and effects in the above-mentioned embodiments of the present disclosure, which will not be elaborated here.

[0156] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0157] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. Image data annotation method, It is characterized in that The method comprises: Acquire an image sequence of a vehicle passing through an intersection and trajectory information of the vehicle; Determining the driving intention and driving process information of the vehicle according to the trajectory information; Annotation information is generated according to the driving intention and driving process information of the vehicle, and the image sequence is annotated according to the annotation information.

2. The image data annotation method according to claim 1, It is characterized in that The trajectory information of the vehicle includes the driving state information of the vehicle; the driving intention of the vehicle includes any one of turning left, going straight, turning right, and making a U-turn.

3. The image data annotation method according to claim 1, It is characterized in that The image sequence consists of N images, wherein at least one image among the N images includes an image of a traffic light, and N is a positive integer.

4. Model training method, It is characterized in that The method comprises: Acquire a training sample set, wherein the training sample set includes a plurality of annotated image sequences, each of which includes at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotation method according to any one of claims 1 to 3; The training sample set is used to train the machine learning model to be trained to obtain a pre-trained machine learning model.

5. The model training method according to claim 4, It is characterized in that After obtaining the training sample set, the method further includes: The image data in the training sample set is standardized, and the machine learning model to be trained is trained based on the standardized training sample set.

6. The model training method according to claim 4, It is characterized in that The step of training the machine learning model to be trained by using the training sample set to obtain the pre-trained machine learning model includes: Based on the labeled image sequence in the training sample set, the machine learning model to be trained is trained using a supervised learning method to obtain the pre-trained machine learning model.

7. Traffic light recognition method, It is characterized in that The method comprises: Obtain a sequence of images to be predicted of the target vehicle; The image sequence to be predicted is input into a pre-trained machine learning model, and a residual network module in the machine learning model performs feature extraction on the image sequence to be predicted to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in any one of claims 4 to 6; Inputting the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector; The second feature vector is input into a classification module in the machine learning model, and the classification module identifies the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

8. The signal light recognition method according to claim 7, It is characterized in that The step of obtaining a sequence of images to be predicted of the target vehicle includes: Based on the image acquisition device installed on the target vehicle, obtaining the driving image video of the target vehicle; The driving image video is subjected to frame extraction processing at a preset frequency to obtain the image sequence to be predicted.

9. The signal light recognition method according to claim 7, It is characterized in that The image sequence to be predicted is N images collected by the target vehicle during driving and used to identify the state of the traffic light, where N is a positive integer.

10. Vehicle start-stop control method, It is characterized in that The method comprises: Obtain target driving intention of target vehicle and location information of preset stop line; When the signal light recognition result corresponding to the target driving intention is a red light, controlling the target vehicle to stop driving at the position corresponding to the preset stop line, wherein the signal light recognition result is obtained by the signal light recognition method described in any one of claims 7 to 9; When the signal light recognition result corresponding to the target driving intention is a green light, the target vehicle is controlled to continue driving at the position corresponding to the preset stop line.

11. The vehicle start-stop control method according to claim 10, It is characterized in that The position information of the preset stop line is determined based on a preset stop line recognition model and internal and external parameters of the image acquisition device.

12. Data annotation device, It is characterized in that The device comprises: An image sequence acquisition module, used to acquire an image sequence when a vehicle passes through an intersection and trajectory information of the vehicle; The image sequence annotation module is used to determine the driving intention and driving process information of the vehicle according to the trajectory information; generate annotation information according to the driving intention and driving process information of the vehicle, and annotate the image sequence according to the annotation information.

13. Model training device, It is characterized in that The device comprises: A sample set acquisition module, used to acquire a training sample set, wherein the training sample set includes a plurality of annotated image sequences, each of which includes at least one image data, wherein the plurality of annotated image sequences are obtained by the image data annotation method according to any one of claims 1 to 3 above; The model training module is used to train the machine learning model to be trained using the training sample set to obtain a pre-trained machine learning model. 14.Signal light recognition device, It is characterized in that The device comprises: A data acquisition module, used to acquire a sequence of images to be predicted of a target vehicle; A feature extraction module, used for inputting the image sequence to be predicted into a pre-trained machine learning model, and extracting features of the image sequence to be predicted by a residual network module in the machine learning model to obtain a first feature vector of a preset dimension, wherein the pre-trained machine learning model is obtained by the model training method described in any one of claims 4 to 6; An encoding processing module, used for inputting the first feature vector into the encoding processing module in the machine learning model for encoding processing to obtain a second feature vector; A traffic light recognition module is used to input the second feature vector into a classification module in the machine learning model, and the classification module recognizes the state of the traffic light according to the second feature vector to obtain a traffic light recognition result.

15. Vehicle control device, It is characterized in that The device comprises: An information acquisition module is used to obtain the target driving intention of the target vehicle and the position information of the preset stop line; A first control module, configured to control the target vehicle to stop driving at a position corresponding to the preset stop line when the signal light recognition result corresponding to the target driving intention is a red light, wherein the signal light recognition result is obtained by the signal light recognition method described in any one of claims 7 to 9; The second control module is used to control the target vehicle to continue driving at the position corresponding to the preset stop line when the signal light recognition result corresponding to the target driving intention is green.

16. An electronic device, It is characterized in that include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 11.

17. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

18. A vehicle, It is characterized in that Comprising the electronic device as claimed in claim 16.

Images