Model training method, vehicle state determination method, device, equipment and vehicle
By combining feature extraction and loss function training of multiple car light state recognition modules, the problem of low vehicle state recognition efficiency and accuracy in the prior art is solved, efficient identification and accuracy of multiple car light states is achieved, and the safety of autonomous driving is improved.
Patent Information
- Application Number
- CN202410207131.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-26
- Publication Date
- 2025-08-26
AI Technical Summary
In the prior art, the efficiency and accuracy of identifying vehicle status by constructing multiple car light recognition models is low, and information sharing cannot be achieved, resulting in low efficiency and accuracy of car light status recognition.
A model training method is adopted, using the feature extraction module of the first car light state recognition model and multiple car light state recognition modules, combined with the car light characteristics, the model is trained through the loss function value to achieve simultaneous recognition of multiple car light states, reducing the number of models, improving training efficiency and recognition accuracy.
It improves the efficiency and accuracy of vehicle status recognition, enhances the generalization ability and robustness of the model, and ensures the safety of autonomous vehicles.
Smart Images
Figure CN120544148A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of autonomous driving technology, and in particular to a model training method, a vehicle state determination method, an apparatus, a device, and a vehicle. Background Art
[0002] With the development of technology, more and more vehicles are equipped with autonomous driving capabilities. To avoid traffic accidents, vehicles in autonomous driving need to accurately identify the status of other vehicles. The status of headlights is an important indicator of vehicle status. Therefore, the status of other vehicles can be identified by identifying the status of their lights.
[0003] Currently, the main method is to build multiple headlight recognition models. Different headlight recognition models are used to identify the status of different headlights. Then, the recognition results of each headlight recognition model are combined to obtain the vehicle status. The efficiency and accuracy are both low. Summary of the Invention
[0004] The embodiments of the present application provide a model training method, a vehicle status determination method, an apparatus, a device, and a vehicle, which can improve the recognition efficiency and accuracy of the vehicle status.
[0005] In a first aspect, an embodiment of the present application provides a model training method, comprising:
[0006] Obtain N training images, where the training images include vehicles, and N is an integer greater than 1;
[0007] Inputting the training image into the feature extraction module of the first headlight state recognition model to obtain headlight features of the training image;
[0008] Inputting the headlight feature into M headlight state recognition modules of a first headlight state recognition model to obtain a first headlight state corresponding to each headlight state recognition module, where M is an integer greater than 1, and each headlight state recognition module is used to identify a different headlight state;
[0009] For each first headlight state, determining a first loss function value based on the first headlight state and a corresponding reference headlight state;
[0010] Based on the first loss function value of each first headlight state, a first headlight state recognition model is trained to obtain a second headlight state recognition model.
[0011] In a second aspect, an embodiment of the present application provides a method for determining a vehicle state, comprising:
[0012] Collect vehicle images;
[0013] Inputting the vehicle image into the second headlight state recognition model to obtain the headlight state of the vehicle;
[0014] Use the front and rear detection model to detect the front and rear of the vehicle image and obtain the front and rear detection results of the vehicle;
[0015] The driving status of the vehicle is determined based on the front and rear detection results of the vehicle and the status of the vehicle's lights.
[0016] In a third aspect, an embodiment of the present application provides a model training device, comprising:
[0017] An acquisition module is used to acquire N training images, where the training images include vehicles, and N is an integer greater than 1;
[0018] A feature extraction module, configured to input the training image into the feature extraction module of the first headlight state recognition model to obtain headlight features of the training image;
[0019] a headlight state recognition module, configured to input headlight features into M headlight state recognition modules of a first headlight state recognition model to obtain a first headlight state corresponding to each headlight state recognition module, where M is an integer greater than 1, and each headlight state recognition module is configured to recognize a different headlight state;
[0020] a determination module, configured to determine, for each first vehicle light state, a first loss function value based on the first vehicle light state and a corresponding reference vehicle light state;
[0021] The training module is used to train the first headlight state recognition model based on the first loss function value of each first headlight state to obtain the second headlight state recognition model.
[0022] In a fourth aspect, an embodiment of the present application provides a vehicle state determination device, comprising:
[0023] An acquisition module, used for acquiring vehicle images;
[0024] A vehicle state recognition module, configured to input a vehicle image into a second headlight state recognition model to obtain a headlight state of the vehicle;
[0025] The detection module is used to detect the front and rear of the vehicle image using the front and rear detection model to obtain the front and rear detection results of the vehicle;
[0026] The determination module is used to determine the driving state of the vehicle based on the detection results of the front and rear ends of the vehicle and the state of the vehicle's lights.
[0027] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method as described in the first aspect or the method as described in the second aspect is implemented.
[0028] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect or the method described in the second aspect.
[0029] In a seventh aspect, an embodiment of the present application provides a storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.
[0030] In an eighth aspect, an embodiment of the present application provides a vehicle, which includes a model training device as described in the third aspect; or a vehicle state determination device as described in the fourth aspect; or an electronic device as described in the fifth aspect; or a computer program product as described in the sixth aspect; or a storage medium as described in the seventh aspect.
[0031] The embodiment of the present application utilizes a feature extraction module of the first headlight state recognition model to extract headlight features from a training image. Using multiple headlight state recognition modules of the first headlight state recognition model, combined with the headlight features, the states of different headlights can be obtained. That is, the first headlight state recognition model of the embodiment of the present application can simultaneously recognize the states of multiple headlights, eliminating the need to construct a corresponding headlight state recognition model for each headlight type. This improves the model's training efficiency and the efficiency of headlight state recognition. Furthermore, when training the first headlight state recognition model, the embodiment of the present application performs training based on the loss function values of each headlight state. That is, the commonalities and characteristics between the headlight features of different headlights are fully considered. This improves the generalization capability of the first headlight state recognition model and the accuracy of the headlight state recognition results, thereby improving the efficiency and accuracy of vehicle state recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;
[0033] Figure 2 A structural diagram of a first vehicle light state recognition model provided in an embodiment of the present application;
[0034] Figure 3 A structural diagram of a feature extraction module provided in an embodiment of the present application;
[0035] Figure 4 A flow chart of a vehicle status determination method provided in an embodiment of the present application;
[0036] Figure 5 A structural diagram of a model training device provided in an embodiment of the present application;
[0037] Figure 6A structural diagram of a vehicle state determination device provided in an embodiment of the present application;
[0038] Figure 7 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0039] In order to more clearly understand the above-mentioned objectives, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.
[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application can also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present application, not all of the embodiments.
[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.
[0042] To avoid traffic accidents, autonomous vehicles need to accurately identify the status of other vehicles. Headlight status is an important indicator of vehicle status, so the status of other vehicles can be identified by identifying their headlights.
[0043] Currently, the main method is to build multiple headlight recognition models, where different headlight recognition models are used to identify the status of different headlights. Then, the recognition results of each headlight recognition model are combined to obtain the status of each headlight.
[0044] For example, for the left and right brake lights, it is necessary to build left and right brake light state recognition models; for the high-mounted brake lights, it is necessary to build a high-mounted brake light state recognition model; for the left turn signal, it is necessary to build a left turn signal state recognition model; for the right turn signal, it is necessary to build a right turn signal state recognition model.
[0045] Each model needs to be built and trained independently, and information sharing is impossible, resulting in low efficiency and accuracy in recognizing the status of vehicle lights.
[0046] To this end, embodiments of the present application provide a model training method, a vehicle status determination method, an apparatus, a device, and a vehicle, which can improve the efficiency and accuracy of recognizing the status of vehicle lights.
[0047] The following describes the model training method, vehicle state determination method, device, equipment and vehicle provided in the embodiments of the present application through specific examples.
[0048] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is as follows: Figure 1 As shown, the model training method may include the following steps:
[0049] S110: Obtain N training images.
[0050] The training image includes vehicles, and N is an integer greater than 1.
[0051] S120: Input the training image into a feature extraction module of the first vehicle headlight state recognition model to obtain vehicle headlight features of the training image.
[0052] S130: Input the vehicle light characteristics into M vehicle light state recognition modules of the first vehicle light state recognition model to obtain a first vehicle light state corresponding to each vehicle light state recognition module.
[0053] Wherein, M is an integer greater than 1, and each vehicle light state recognition module is used to recognize the state of different vehicle lights.
[0054] S140. For each first headlight state, determine a first loss function value based on the first headlight state and a corresponding reference headlight state.
[0055] S150 : Based on the first loss function value of each first headlight state, train a first headlight state recognition model to obtain a second headlight state recognition model.
[0056] The embodiment of the present application utilizes a feature extraction module of the first headlight state recognition model to extract headlight features from a training image. Using multiple headlight state recognition modules of the first headlight state recognition model, combined with the headlight features, the states of different headlights can be obtained. That is, the first headlight state recognition model of the embodiment of the present application can simultaneously recognize the states of multiple headlights, eliminating the need to construct a corresponding headlight state recognition model for each headlight type. This improves the model's training efficiency and the efficiency of headlight state recognition. Furthermore, when training the first headlight state recognition model, the embodiment of the present application performs training based on the loss function values of each headlight state. That is, the commonalities and characteristics between the headlight features of different headlights are fully considered. This improves the generalization capability of the first headlight state recognition model and the accuracy of the headlight state recognition results, thereby improving the efficiency and accuracy of vehicle state recognition.
[0057] The above steps are explained in detail below:
[0058] In S110, the training image may be an image containing a vehicle. For example, the training image may be a historical image captured by the autonomous vehicle during the autonomous driving process, or an image obtained by processing the historical image. For example, if the historical image contains multiple vehicles, each vehicle may be cropped and the cropped image used as the training image. This allows for better application in autonomous driving scenarios.
[0059] The vehicles in the training images can be vehicles located in front of, behind, or on the sides of the autonomous vehicle. By identifying the headlight status of vehicles in front of, behind, and on the sides of the autonomous vehicle, it helps the autonomous vehicle to identify the driving status of these vehicles and then adopt appropriate autonomous driving strategies to ensure driving safety.
[0060] In S120, the first vehicle light state recognition model is a vehicle light state recognition model before training, and the first vehicle light state recognition model is used to recognize the vehicle light state, which may include on, off, unknown, etc.
[0061] like Figure 2 As shown, the first vehicle headlight state recognition model 200 may include a feature extraction module 201 and M vehicle headlight state recognition modules 202. The feature extraction module 201 is used to extract vehicle headlight features of each training image. The vehicle headlight features may include, for example, the position, brightness, color, etc. of the vehicle headlights.
[0062] The headlight state recognition module 202 is used to identify the headlight state based on the headlight features extracted by the feature extraction module. In the embodiment of the present application, the headlight state recognition module 202 can include multiple modules, that is, it can simultaneously identify the states of multiple headlights, thereby reducing the number of models and improving the training efficiency of the models. Figure 2 Take M=4 as an example.
[0063] Exemplarily, the vehicle light status recognition module may include at least two of the following:
[0064] Left light status recognition module, right light status recognition module, left and right brake light status recognition module, high-mounted brake light status recognition module;
[0065] Among them, the left light status recognition module is used to identify the status of the left light, the right light status recognition module is used to identify the status of the left light, the left and right brake light status recognition modules are used to identify the status of the left and right brake lights, and the high-mounted brake light status recognition module is used to identify the status of the high-mounted brake light. The left light and the right light are respectively one type of turn signal.
[0066] It can be understood that the state of the turn signal is related to the position of the turn signal. For example, for the front of the vehicle, if the left turn signal of the vehicle is on, the corresponding right light is on in the training image. For another example, for the rear of the vehicle, if the left turn signal of the vehicle is on, the corresponding left light is on in the training image.
[0067] In order to simplify the training process of the first vehicle light state recognition model and improve the training effect, the embodiment of the present application uniformly defines the turn signals as left lights and right lights. Subsequently, the vehicle's driving state can be determined by further combining the vehicle light states of the left lights and right lights and the front and rear detection results of the vehicle.
[0068] For example, the left light state recognition module is used to recognize the state of the left light in the training image. Similarly, the right light state recognition module is used to recognize the state of the right light in the training image.
[0069] For example, the left side light (left_side_light), right side light (right_side_light), and high-mounted brake light (high_side_brake_light_status) each have three corresponding states: on, off, and unknown. The left and right brake lights (left_right_side_brake_light_status) can also have the position lights on. The "unknown" state applies to situations where the vehicle cannot be determined, such as due to obstruction.
[0070] In some embodiments, the vehicle light status recognition module of the embodiment of the present application may include the above four modules at the same time, that is, it can simultaneously recognize the vehicle light status of the left side light, the right side light, the left and right brake lights, and the high-mounted brake lights, which can not only save the model construction and training costs, but also improve the model training efficiency and training effect, thereby improving the recognition efficiency of the vehicle light status and the accuracy of the recognition results.
[0071] The embodiments of the present application do not limit the specific structures of the feature extraction module and the vehicle light status recognition module. Any structure that can meet the above requirements can be applied to the embodiments of the present application.
[0072] For example, the feature extraction module can adopt the Swin Transformer model. The Swin Transformer model is a Transformer-based model that adopts a novel structure. Therefore, the Swin Transformer can also be called a "Swin Block".
[0073] Swin Block can be composed of four sub-layers: cross-window attention layer, local attention layer, perceptron layer and residual connection layer. The cross-window attention layer and local attention layer are used to extract global features and local features of the training image respectively, the perceptron layer is used to process global features and local features, and the residual connection layer is used to pass the original features of the training image to subsequent layers.
[0074] For example, the structure of the feature extraction module 201 in the embodiment of the present application can be seen in Figure 3 The feature extraction module 201 may include an image block segmentation layer Patch Partition 300 and four feature extraction layers.
[0075] The image block segmentation layer is used to segment the input training image into blocks, for example, the training image can be segmented into blocks of 56*56*48. For example, the size of the training image is 224*224*3.
[0076] Each feature extraction layer consists of a linear embedding layer and a certain number of Swin blocks. For example, the first feature extraction layer 301 includes a first linear embedding layer and two Swin blocks. The second feature extraction layer 302 includes a second linear embedding layer and two Swin blocks. The third feature extraction layer 303 includes a third linear embedding layer and 18 Swin blocks. The fourth feature extraction layer 304 includes a fourth linear embedding layer and two Swin blocks. The Swin blocks in each layer are connected in series, that is, the output of the previous Swin block serves as the input of the next Swin block.
[0077] In some embodiments, in order to improve the training efficiency of the first headlight state recognition model, the feature extraction module 201 may further include a global average pooling (GAP) layer, which is used to reduce the dimension of the headlight features. The input of the GAP layer is the output of the fourth feature extraction layer 304.
[0078] Through the above-mentioned feature extraction module, deep features of the training image can be extracted, thereby improving the training accuracy of the first vehicle light state recognition model.
[0079] The vehicle light state recognition module may include, for example, a fully connected layer and a softmax normalization layer.
[0080] In S130 , after the vehicle light features are obtained, the vehicle light features may be input into M vehicle light state recognition modules respectively, and each vehicle light state recognition module processes the vehicle light features respectively to obtain a corresponding first vehicle light state.
[0081] In S140, for each first headlight state, a first loss function value may be determined based on the first headlight state and a corresponding reference headlight state, in combination with a loss function. The reference headlight state is the actual headlight state of the vehicle in the training image. The loss function may be set as needed.
[0082] In S150 , after the first loss function values corresponding to the first headlight states are determined, the model parameters of the first headlight state recognition model may be adjusted based on the first loss function values to obtain the second headlight state recognition model.
[0083] Exemplarily, a weighted summation may be performed on each first loss function value to obtain a second loss function value;
[0084] Adjusting model parameters of the first vehicle light state recognition model based on the second loss function value until all first loss function values meet a convergence condition;
[0085] The first vehicle light state recognition model when all first loss function values satisfy the convergence condition is determined as the second vehicle light state recognition model.
[0086] In an embodiment of the present application, the weights of the first loss function values may be the same or different. For example, the weights of the first loss function values may all be 1, that is, the second loss function value may be obtained by adding up the first loss function values.
[0087] Based on the second loss function value, the model parameters of the first vehicle light state recognition model may be adjusted until all first loss function values satisfy a convergence condition.
[0088] For example, assuming that a first loss function value of a first headlight state first satisfies the convergence condition, when the model parameters of the first headlight state recognition model are subsequently adjusted based on the second loss function value, it is necessary to make other first loss function values meet the convergence condition, and at the same time, the fluctuation of the first loss function value that has met the convergence condition should be kept as small as possible. That is, the output results of each headlight state recognition module need to be taken into account at the same time, thereby improving the robustness of the first headlight state recognition model to the headlight states of different headlights.
[0089] The embodiment of the present application uses the loss function value of each headlight state to train the first headlight state recognition model, which can improve the generalization ability and robustness of the first headlight state recognition model.
[0090] In some embodiments, the above S110 may include the following steps:
[0091] Acquire N1 initial images, each of which includes at least one vehicle;
[0092] Crop the vehicles in each initial image to obtain N2 cropped images;
[0093] Perform data augmentation on each cropped image to obtain N training images, where N is greater than N2.
[0094] The initial image may contain at least one vehicle. During actual training, if the initial image contains multiple vehicles, each vehicle can be cropped to obtain an image containing a single vehicle. In other words, the cropped image is an image containing a single vehicle.
[0095] In order to improve the diversity of training samples, data enhancement processing can be performed on each cropped image, and the data-enhanced image and the cropped image are determined as training images to train the first vehicle light state recognition model.
[0096] The embodiments of the present application do not limit the specific method of data enhancement. For example, the cropped image can be enhanced by changing the spatial position features, geometric features, chromaticity (Red-Green-Blue, RGB) features, etc. of the image.
[0097] For example, when a cropped image includes both left and right lights, the diversity of samples can be increased by changing the spatial positional features of the image. For example, the left light can be horizontally flipped around the center point between the left and right lights to obtain samples of the right light. Similarly, the right light can be horizontally flipped to obtain samples of the left light. This can enhance the first vehicle light state recognition model's learning of the left and right lights.
[0098] It is understandable that the size of the cropped image actually obtained may be inconsistent. For this case, for example, an area threshold can be set. If the product of the length and width of the cropped image is greater than the first area threshold, the area of the cropped image can be reduced by 1 / 3 to 1 / 2. If the product of the length and width of the cropped image is less than or equal to the first area threshold and greater than the second area threshold, the area of the cropped image can be reduced by 2 / 3. If the product of the length and width of the cropped image is less than or equal to the second area threshold, the size of the cropped image can be kept unchanged. The above reduction ratio is just one example and can be adjusted as needed. In this way, the number of small-size samples can be increased without reducing image information.
[0099] For example, the diversity of samples can be increased by changing the RGB values of the image. For example, random grayscale, color jitter, and Gaussian blur can be used to process the cropped image.
[0100] Random grayscale can enhance the robustness of the first headlight state recognition model to color information and reduce its reliance on color. Color dithering can randomly modify the brightness, contrast, saturation, and hue of the cropped image, changing the color of the cropped image and enhancing the robustness of the first headlight state recognition model to different colors. Gaussian blur can simulate the blur of the cropped image, improving the adaptability and robustness of the first headlight state recognition model to blurred images.
[0101] Performing random grayscale conversion, color dithering, and Gaussian blur processing on the cropped image simultaneously can avoid overfitting of the first headlight state recognition model.
[0102] For example, when performing random grayscale, color dithering and Gaussian blur processing on the cropped image, the above processing can be performed only on part of the cropped image to retain the original features of the part of the cropped image, thereby avoiding the loss of certain detail information and affecting the training effect of the first vehicle light state recognition model.
[0103] It should be noted that when performing data augmentation processing on the cropped image, the enhancement intensity should be controlled to avoid excessive perturbation of the cropped image and affecting the training effect of the first headlight state recognition model.
[0104] By performing data enhancement on the cropped image, the adaptability of the first vehicle light state recognition model to color, blur and other conditions can be enhanced, thereby improving the generalization ability and robustness of the first vehicle light state recognition model.
[0105] The first headlight state recognition model of the embodiment of the present application implements multi-task learning, which can process multiple related tasks at the same time, thereby improving learning efficiency. Moreover, multi-task learning can share knowledge with the help of the correlation between tasks, thereby improving the performance of the first headlight state recognition model. In addition, multi-task learning also helps to improve the generalization ability and robustness of the first headlight state recognition model, thereby improving the recognition efficiency of the headlight state and the accuracy of the recognition results.
[0106] After the vehicle light status is identified, the vehicle can be controlled to automatically drive based on the identified vehicle light status. Based on this, in some embodiments, the present application also provides a vehicle state determination method, such as Figure 4 As shown, the vehicle state determination method may include the following steps:
[0107] S410: Collect vehicle images.
[0108] S420: Input the vehicle image into a second vehicle light state recognition model to obtain the vehicle light state.
[0109] S430: Detect the front and rear of the vehicle image using the front and rear detection model to obtain detection results of the front and rear of the vehicle.
[0110] S440: Determine the driving state of the vehicle based on the front and rear detection results of the vehicle and the status of the vehicle's lights.
[0111] The above steps are explained in detail below:
[0112] In S410, the vehicle can be one or more vehicles located in front of, behind, or to the side of the vehicle. When there are multiple vehicles in front of, behind, or to the side of the vehicle, each vehicle is handled in the same manner. During the autonomous driving process, by obtaining the driving status of other vehicles, traffic accidents can be effectively avoided.
[0113] In S420, the second vehicle light state recognition model may be trained using the model training method described in the above embodiment. For example, if the vehicle image contains only one vehicle, the vehicle image may be directly input into the second vehicle light state recognition model described in the above embodiment to determine the vehicle light state in the image. For details on vehicle light states, please refer to the above embodiment.
[0114] For example, if the vehicle image contains multiple vehicles, the vehicle image can be cropped to obtain a cropped image containing a single vehicle. At this time, the cropped image can be input into the second headlight state recognition model to obtain the headlight state of the vehicle in the cropped image.
[0115] It is understood that for the same vehicle light status, for example, if the left light is on, if the vehicle image includes the front of another vehicle, then the left light is on, indicating that the other vehicle is about to turn right. If the vehicle image includes the rear of another vehicle, then the left light is on, indicating that the other vehicle is about to turn left. In other words, the driving status of another vehicle is not only related to the status of the headlights, but also to the front and rear of the vehicle. The driving status of a vehicle can include, for example, turning left, turning right, or going straight.
[0116] In order to accurately determine the driving status of other vehicles, for example, the front and rear ends of each vehicle image may be detected to determine whether each vehicle image contains the front or the rear end of the vehicle.
[0117] For example, the vehicle image may be detected using a vehicle front and rear detection model to determine whether the vehicle image contains the vehicle front or the vehicle rear.
[0118] For example, the vehicle front and rear detection model may adopt a target detection model (DETR with Improved de Noising anchOr boxes, DINO), or may adopt other forms, which is not limited in the embodiments of the present application.
[0119] In S430, the driving status of other vehicles can be determined based on the front and rear vehicle detection results and the status of the vehicle lights. In some embodiments, the driving status of other vehicles can be used to control the autonomous driving of the own vehicle. In some embodiments, the driving status of other vehicles can also be used to provide warnings to the own vehicle.
[0120] For example, with the right-side light on, if the front-to-rear detection result is the rear, it means that the other vehicle is about to turn right, that is, the driving status of the other vehicle is turning right; if the front-to-rear detection result is the front, it means that the other vehicle is about to turn left, that is, the driving status of the other vehicle is turning left, that is, for the front, the driving status of the other vehicle is in opposite logic to the headlight status, and for the rear, the driving status of the other vehicle is in the same logic as the headlight status.
[0121] The second headlight state recognition model can simultaneously identify the headlight states of multiple headlights, thereby improving the efficiency and accuracy of headlight recognition. Combined with the front and rear detection results, the driving states of other vehicles can be accurately judged to ensure driving safety.
[0122] Based on the same inventive concept, the present application embodiment also provides a model training device. Figure 5 The model training device provided in the embodiment of the present application is described in detail.
[0123] like Figure 5 As shown, the model training device may include:
[0124] An acquisition module 501 is configured to acquire N training images, wherein the training images include vehicles, and N is an integer greater than 1;
[0125] A feature extraction module 502 is configured to input the training image into the feature extraction module of the first vehicle headlight state recognition model to obtain vehicle headlight features of the training image;
[0126] a headlight state recognition module 503 configured to input headlight features into M headlight state recognition modules of a first headlight state recognition model to obtain a first headlight state corresponding to each headlight state recognition module, where M is an integer greater than 1, and each headlight state recognition module is configured to recognize a different headlight state;
[0127] a determination module 504 for determining, for each first vehicle light state, a first loss function value based on the first vehicle light state and a corresponding reference vehicle light state;
[0128] The training module 505 is configured to train the first vehicle light state recognition model based on the first loss function value of each first vehicle light state to obtain a second vehicle light state recognition model.
[0129] The embodiment of the present application utilizes a feature extraction module of a first headlight state recognition model to extract headlight features from a training image. Multiple headlight state recognition modules of the first headlight state recognition model, combined with the headlight features, can be used to determine the states of different headlights. This means that the first headlight state recognition model of the embodiment of the present application can simultaneously recognize the states of multiple headlights, eliminating the need to construct a corresponding headlight state recognition model for each headlight type. This improves the efficiency of model training and the efficiency of headlight state recognition. Furthermore, when training the first headlight state recognition model, the embodiment of the present application performs training based on the loss function values for each headlight state. This means that the commonalities and characteristics between the headlight features of different headlights are fully considered, thereby improving the generalization capability of the first headlight state recognition model and the accuracy of the headlight state recognition results.
[0130] In some embodiments, the training module 505 is specifically configured to:
[0131] Perform weighted summation on the first loss function values to obtain the second loss function value;
[0132] Adjusting model parameters of the first vehicle light state recognition model based on the second loss function value until all first loss function values meet a convergence condition;
[0133] The first vehicle light state recognition model when all first loss function values satisfy the convergence condition is determined as the second vehicle light state recognition model.
[0134] In some embodiments, the acquisition module 501 is specifically configured to:
[0135] Acquire N1 initial images, each of which includes at least one vehicle;
[0136] The model training device may further include:
[0137] A cropping module, used to crop the vehicles in each initial image to obtain N2 cropped images;
[0138] The processing module is used to perform data enhancement processing on each cropped image to obtain N training images, where N is greater than N2.
[0139] In some embodiments, the M vehicle light state recognition modules include at least two of the following:
[0140] Left light status recognition module, right light status recognition module, left and right brake light status recognition module, high-mounted brake light status recognition module;
[0141] Among them, the left light status recognition module is used to identify the status of the left light, the right light status recognition module is used to identify the status of the left light, the left and right brake light status recognition modules are used to identify the status of the left and right brake lights, and the high-mounted brake light status recognition module is used to identify the status of the high-mounted brake light. The left light and the right light are respectively one type of turn signal.
[0142] Figure 5 Each module in the device shown has the function of realizing Figure 1 The functions of each step in the process can achieve the corresponding technical effects, so for the sake of brevity, they will not be described here in detail.
[0143] Based on the same inventive concept, the embodiment of the present application also provides a vehicle state determination device, which is described below in conjunction with Figure 6 The vehicle state determination device provided in the embodiment of the present application is described in detail.
[0144] like Figure 6 As shown, the vehicle state determination device may include:
[0145] Acquisition module 601, used to acquire vehicle images;
[0146] The vehicle state recognition module 602 is used to input the vehicle image into the second headlight state recognition model to obtain the vehicle headlight state;
[0147] The detection module 603 is used to detect the front and rear of the vehicle image using the front and rear detection model to obtain the front and rear detection results of the vehicle;
[0148] The determination module 604 is used to determine the driving state of the vehicle based on the detection results of the front and rear ends of the vehicle and the state of the vehicle lights.
[0149] The second headlight state recognition model can simultaneously identify the headlight states of multiple headlights, thereby improving the efficiency and accuracy of headlight recognition. Combined with the front and rear detection results, the driving states of other vehicles can be accurately judged to ensure driving safety.
[0150] Figure 6 Each module in the device shown has the function of realizing Figure 4 The functions of each step in the process can achieve the corresponding technical effects, so for the sake of brevity, they will not be described here in detail.
[0151] Based on the same inventive concept, the present application embodiment also provides an electronic device. Figure 7 The electronic device provided in the embodiments of the present application is described in detail.
[0152] like Figure 7 As shown, the electronic device may include a processor 710 and a memory 720 for storing computer program instructions.
[0153] The processor 710 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0154] The memory 720 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 720 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In one example, the memory 720 may include a removable or non-removable (or fixed) medium, or the memory 720 may be a non-volatile solid-state memory. In one example, the memory 720 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0155] The processor 710 reads and executes computer program instructions stored in the memory 720 to implement Figure 1 or Figure 4 The method in the embodiment shown in FIG. Figure 1 or Figure 4The corresponding technical effects achieved by executing the method in the illustrated embodiment are not described in detail here for the sake of brevity.
[0156] In one example, the electronic device may further include a communication interface 730 and a bus 740. Figure 7 As shown, the processor 710 , the memory 720 , and the communication interface 730 are connected via a bus 740 and communicate with each other.
[0157] The communication interface 730 is mainly used to implement communication between various modules, devices and / or equipment in the embodiments of the present application.
[0158] Bus 740 includes hardware, software or both, and each component of electronic equipment is coupled to each other.For example, but not limitation, bus 740 may include accelerated graphics end (Accelerated Graphics Port, AGP) or other graphics buses, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), hyper transport (Hyper Transport, HT) interconnection, industry standard architecture (Industry Standard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 740 may include one or more buses. Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.
[0159] After acquiring N training images, the electronic device can execute the model training method in the embodiment of the present application, thereby realizing the combination of Figure 1 The model training method described and Figure 5 Model training device described. Alternatively,
[0160] After collecting the vehicle image, the electronic device can execute the vehicle state determination method in the embodiment of the present application, thereby realizing the combination of Figure 4 The vehicle state determination method described and Figure 6 A vehicle state determination device is described.
[0161] Based on the same inventive concept, an embodiment of the present application provides a computer program product, including a computer program, which is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment or the above-mentioned vehicle state determination method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0162] Based on the same inventive concept, an embodiment of the present application also provides a storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the model training method described in the above embodiment or the vehicle state determination method described in the above embodiment is implemented.
[0163] Based on the same inventive concept, an embodiment of the present application also provides a vehicle, which includes a model training device as described in the above embodiment; or a vehicle state determination device as described in the above embodiment; or an electronic device as described in the above embodiment; or a storage medium as described in the above embodiment.
[0164] It should be understood that the above-mentioned specific embodiments of the present application are merely illustrative or explain the principles of the present application and do not constitute a limitation of the present application. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present application should be included in the scope of protection of the present application. In addition, the claims attached hereto are intended to cover all variations and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.
[0165] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0166] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.
[0167] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.
Claims
1. A model training method, characterized in that: include: Obtaining N training images, wherein the training images include vehicles, where N is an integer greater than 1; Inputting the training image into a feature extraction module of a first vehicle headlight state recognition model to obtain vehicle headlight features of the training image; Inputting the headlight feature into M headlight state recognition modules of the first headlight state recognition model to obtain a first headlight state corresponding to each headlight state recognition module, where M is an integer greater than 1, and each headlight state recognition module is used to identify a different headlight state; For each of the first headlight states, determining a first loss function value based on the first headlight state and a corresponding reference headlight state; Based on the first loss function value of each of the first headlight states, the first headlight state recognition model is trained to obtain a second headlight state recognition model.
2. The method according to claim 1, characterized in that The training of the first headlight state recognition model based on the first loss function value of each of the first headlight states to obtain a second headlight state recognition model includes: Performing a weighted summation on the first loss function values to obtain a second loss function value; Adjusting model parameters of the first vehicle light state recognition model based on the second loss function value until all first loss function values meet a convergence condition; The first vehicle light state recognition model when all the first loss function values satisfy the convergence condition is determined as the second vehicle light state recognition model.
3. The method according to claim 1 or 2, characterized in that The obtaining of N training images includes: Acquire N1 initial images, each of which includes at least one vehicle; Cropping the vehicle in each of the initial images to obtain N2 cropped images; Data enhancement processing is performed on each of the cropped images to obtain N training images, where N is greater than N2.
4. The method according to claim 1 or 2, characterized in that The M vehicle light status recognition modules include at least two of the following: Left light status recognition module, right light status recognition module, left and right brake light status recognition module, high-mounted brake light status recognition module; Among them, the left light status recognition module is used to identify the status of the left light, the right light status recognition module is used to identify the status of the left light, the left and right brake light status recognition modules are used to identify the status of the left and right brake lights, and the high-mounted brake light status recognition module is used to identify the status of the high-mounted brake light. The left light and the right light are respectively one type of turn signals.
5. A method for determining a vehicle state, characterized in that: include: Collect vehicle images; Inputting the vehicle image into a second vehicle light state recognition model to obtain the vehicle light state; Performing front and rear detection on the vehicle image using a front and rear detection model to obtain front and rear detection results of the vehicle; The driving state of the vehicle is determined based on the front and rear detection results of the vehicle and the state of the vehicle lights.
6. A model training device, characterized in that: include: An acquisition module, configured to acquire N training images, wherein the training images include vehicles, where N is an integer greater than 1; a feature extraction module, configured to input the training image into the feature extraction module of the first headlight state recognition model to obtain headlight features of the training image; a headlight state recognition module, configured to input the headlight feature into M headlight state recognition modules of the first headlight state recognition model to obtain a first headlight state corresponding to each headlight state recognition module, where M is an integer greater than 1, and each headlight state recognition module is configured to recognize a different headlight state; a determination module, configured to determine, for each first headlight state, a first loss function value based on the first headlight state and a corresponding reference headlight state; A training module is used to train the first headlight state recognition model based on the first loss function value of each first headlight state to obtain a second headlight state recognition model.
7. A vehicle state determination device, characterized in that: include: An acquisition module, used for acquiring vehicle images; a vehicle state recognition module, configured to input the vehicle image into a second headlight state recognition model to obtain the vehicle headlight state; A detection module, configured to detect the front and rear ends of the vehicle image using a front and rear detection model to obtain front and rear end detection results of the vehicle; The determination module is used to determine the driving state of the vehicle based on the detection results of the front and rear ends of the vehicle and the state of the vehicle lights.
8. An electronic device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method according to any one of claims 1 to 4 or the method according to claim 5 is implemented.
9. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4 or the method according to claim 5.
10. A vehicle, characterized in that: The vehicle comprises the model training device according to claim 6; or the vehicle state determination device according to claim 7; or the electronic device according to claim 8; or the computer program product according to claim 9.