Vehicle signal light recognition method, recognition model training method and related equipment

Abstract

The present application discloses a vehicle signal light recognition method, a recognition model training method and related equipment. The method comprises: obtaining a target image of a vehicle; detecting the signal light of a target part of the target image to obtain at least one target part signal light area; processing at least one target part signal light area using a recognition model to obtain at least one corresponding target part signal light state information, wherein the recognition model is obtained by training using at least two different types of target part signal light areas. The above scheme can improve the accuracy of the recognition of the state information of the target part signal light.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for recognizing vehicle signal lights, a method for training a recognition model of vehicle signal lights, an electronic device, and a storage device. Background Art

[0002] With the development of social economy and the improvement of people's living standards, the number of motor vehicles in the country has increased year by year, and the safe driving of vehicles on urban roads has become increasingly important. Cars on the road send driving signals through signal lights or headlights to warn other vehicles.

[0003] At present, since the vehicle's signal lights are affected by the environment, such as strong light, signal light obstruction and other problems, it is impossible to accurately identify the vehicle's signal lights, which affects the vehicle's driving safety and easily causes traffic accidents and other problems. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a vehicle signal light recognition method, a recognition model training method and related equipment, which can improve the accuracy of the status information recognition of the signal light at the target location.

[0005] In order to solve the above problems, the first aspect of the present application provides a method for identifying vehicle signal lights, the method comprising: acquiring a target image of a vehicle; detecting signal lights at a target location on the target image to obtain at least one target location signal light area; processing at least one target location signal light area using a recognition model to obtain status information of at least one corresponding target location signal light, wherein the recognition model is trained using at least two different types of target location signal light areas.

[0006] In order to solve the above problems, the second aspect of the present application provides a training method for a vehicle signal light recognition model, the method comprising: obtaining a sample image of a sample vehicle; wherein the sample image comprises N part sample images of N part signal light areas corresponding to different types of signal lights, N being an integer greater than 1; using the recognition model to process the N part sample images, and respectively obtaining part prediction state information and combined prediction state information corresponding to each of the N part signal light areas, wherein the combined prediction state information is obtained based on the N part signal light areas; and adjusting the parameters of the recognition model based on the part prediction state information and the combined prediction state information to obtain a trained recognition model.

[0007] In order to solve the above-mentioned problem, the third aspect of the present application provides an electronic device, which includes a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement any step of the above-mentioned vehicle signal light recognition method and vehicle signal light recognition model training method.

[0008] In order to solve the above problems, the fourth aspect of the present application provides a storage device, which stores program data that can be executed by a processor, and the program data is used to implement any step of the above-mentioned vehicle signal light recognition method and vehicle signal light recognition model training method.

[0009] The above scheme obtains a target image of the vehicle, detects the signal lights at the target part of the target image, and obtains at least one target part signal light area. Since the recognition model is trained using at least two different types of target part signal light areas, the recognition model can comprehensively learn the characteristics of at least two different types of target part signal light areas, and use the recognition model to process at least one target part signal light area to obtain the status information of at least one corresponding target part signal light, which can improve the accuracy of recognition of the status information of the target part signal light. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0011] Figure 1 It is a flow chart of the first embodiment of the vehicle signal light recognition method of the present application;

[0012] Figure 2 is a schematic diagram of an example of an embodiment of a target image of the present application;

[0013] Figure 3 This is a schematic diagram of an example of the signal light area of ​​the first part of the present application;

[0014] Figure 4 This is a schematic diagram of an example of the second signal light area of ​​the present application;

[0015] Figure 5 This application Figure 1 A schematic diagram of a flow chart of an embodiment of step S12;

[0016] Figure 6 It is a structural diagram of an embodiment of the identification model of the present application;

[0017] Figure 7 This application Figure 1 A schematic diagram of a flow chart of another embodiment of step S12;

[0018] Figure 8 It is a structural schematic diagram of another embodiment of the identification model of the present application;

[0019] Fig. 9 It is a structural schematic diagram of another embodiment of the identification model of the present application;

[0020] Fig.10 It is a flow chart of the second embodiment of the vehicle signal light recognition method of the present application;

[0021] Fig.11 It is a flow chart of the third embodiment of the vehicle signal light recognition method of the present application;

[0022] Fig.12 It is a flow chart of an embodiment of a training method for a vehicle signal light recognition model of the present application;

[0023] Fig.13 It is a structural schematic diagram of a first embodiment of a vehicle signal light recognition device of the present application;

[0024] Fig.14 It is a structural schematic diagram of a second embodiment of a vehicle signal light recognition device of the present application;

[0025] Fig.15 It is a structural schematic diagram of an embodiment of a training device for a recognition model of a vehicle signal light of the present application;

[0026] Fig.16 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0027] Fig.17 It is a structural diagram of an embodiment of a storage device of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0030] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] The present application provides the following embodiments, and each embodiment is described in detail below.

[0032] See also Figure 1 , Figure 1 This is a flow chart of the first embodiment of the vehicle signal light recognition method of the present application. The method may include the following steps:

[0033] S11: Acquire a target image of a vehicle.

[0034] The method can be applied to computer equipment, electronic equipment, etc. The electronic equipment may include a vehicle-mounted camera, a surveillance camera, or a smart device connected to a lane camera, etc. The present application is not limited thereto.

[0035] In some embodiments, the target image of the vehicle may be obtained based on an image or image frame taken of the vehicle. For example, the vehicle may be photographed by a camera device to collect a vehicle image or video in real time. The vehicle image or video frame may contain multiple vehicles or one vehicle. Thus, the target image of the vehicle may be obtained based on the vehicle image or video frame. The vehicle may be one or more target vehicles in the vehicle image.

[0036] In some embodiments, the target image may include an image of the rear of the vehicle, and a signal light is provided at the target portion of the rear of the vehicle. The target image may also be an image including the rear portion of the vehicle or the target portion, and the present application does not limit this.

[0037] In some implementations, a vehicle image obtained by photographing a vehicle may be acquired, the vehicle may be detected in the acquired vehicle image, and a target image of the vehicle may be acquired from the vehicle image, so as to obtain a set of target images from the vehicle image, wherein the set of target images may be expressed as: Cars = {C1, C2, ..., C K}, where K is a natural number, K represents the total number of target images, that is, the total number of target images contained in the vehicle image, and the i-th target image C i Contains the coordinate information of the detection box of the target image, denoted as C i =(x i ,y i ,w i ,h i ), x i ,y i ,w i ,h i They can represent the location coordinates, width, and height of the target image respectively.

[0038] S12: Detecting the signal lights of the target part of the target image to obtain at least one signal light area of ​​the target part.

[0039] Since the target image includes a rear image of the vehicle, a signal light is arranged at the rear of the vehicle. The target part may include multiple parts, and in the target image, the signal lights at the multiple parts are detected to obtain at least one target part signal light area.

[0040] In some embodiments, at least one target position signal light area includes N position signal light areas corresponding to different types of signal lights, where N is an integer greater than 1. The signal lights may be classified according to the position of the vehicle where they are located, or according to the function of the signal lights, etc., which is not limited in the present application. Different types of signal lights may correspond to different signal light areas.

[0041] Due to the influence of the shooting angle of the vehicle, the occlusion of the target part in the target image, etc., the signal light areas of some target parts may not be detected. When detecting the signal light of each target part, some signal light areas among the N signal light areas may be detected, and the detected signal light areas are cropped from the target image to obtain at least one signal light area of ​​the target part.

[0042] In some embodiments, the signal light may include a brake light for braking reminder, a turn signal for turning reminder, a fog light for lighting, etc., and this application does not limit this. The following description of this application takes the signal light as a brake light as an example.

[0043] Taking the classification by position as an example, if the signal light is used for brake reminder, type 1 can be the brake lights installed at both ends of the rear of the car, which is a kind of car taillight and can include left brake light and right brake light. Type 2 can be the high-mounted brake light installed on the upper part of the rear of the car.

[0044] In some embodiments, see Figure 2 , when N is 2, the N signal light areas include a first signal light area and a second signal light area. If the signal light is used for brake reminder, the first signal light area is the left brake light area and / or the right brake light area, and the second signal light area is the high-mounted brake light area. When the vehicle receives a brake command, the signal lights corresponding to its left brake light area, right brake light area and / or high-mounted brake light area will light up. The high-mounted brake light area is usually installed on the upper part of the rear of the vehicle, higher than the left brake light area or the right brake light area, and the high-mounted brake light area is less likely to be blocked. In addition, the high-mounted brake light has a fixed shape and a single function, and is easier to identify than the left brake light and right brake light that integrate turn signals and position markers.

[0045] In some embodiments, see Figure 3 and Figure 4 The first signal light area may represent an image cropped from a target image and including the first signal light, and the second signal light area may represent an image cropped from a target image and including the second signal light, but the present application is not limited thereto.

[0046] In some embodiments, when the target part is a brake light area, the brake light area of ​​the target image is detected, and the signal light of the detected target part of each target image is recorded. The at least one target part signal light area detected can be represented as B i{x|x∈{l,r,h}} =(x ix ,y ix ,w ix ,h ix ), where B il , B ir and B ih Respectively represent the coordinate information of the left brake light area, the right brake light area and the high-mounted brake light area.

[0047] S13: Processing at least one target location signal light region using a recognition model to obtain state information of at least one corresponding target location signal light, wherein the recognition model is trained using at least two different types of target location signal light regions.

[0048] The recognition model can be used to classify the status information of at least one target part signal light. The recognition model is obtained by training using two different types of target part signal light areas. Wherein, in the process of training the recognition model, N sample images containing N parts signal light areas (such as sample images of the first part signal light area and the second part signal light area) can be combined for training, so that the trained recognition model can learn the characteristic information of the N parts signal light areas. For example, the sample images of the left brake light area and the right brake light area are respectively combined with the sample images of the high-position brake light area to train the recognition model to obtain a trained recognition model, so that the trained recognition model learns the characteristic information of the first part signal light area and the second part signal light area. In this embodiment, the recognition model and the specific training process of the recognition model can refer to the following embodiment, which is not limited here.

[0049] In some embodiments, the recognition model may be used to process at least one target location signal light area to obtain the state information of at least one corresponding target location signal light. The state information of at least one corresponding target location signal light includes a light state and a non-light state. The light state means that the state of the signal light is recognized as a light state; the non-light state means that the state of the signal light is an off state or the state of the signal light cannot be recognized as a light state.

[0050] In some embodiments, N is 2 for example. If the detected at least one target signal light area includes a first signal light area and a second signal light area, the first signal light area and the second signal light area can be processed using the recognition model to obtain the corresponding state information of the first signal light. For example, if the left brake light area and the high-mounted brake light area are detected, the left brake light area and the high-mounted brake light area can be used as a combined input recognition model for processing to obtain the corresponding state information of the signal light in the left brake light area.

[0051] In some embodiments, if at least one detected target position signal light area only includes the first position signal light area or only includes the second position signal light area, the first position signal light area can be processed using the recognition model to obtain the corresponding status information of the first position signal light, or the second position signal light area can be processed using the recognition model to obtain the corresponding status information of the second position signal light.

[0052] In this embodiment, a target image of a vehicle is acquired and signal lights at a target location are detected on the target image to obtain at least one target location signal light area. Since the recognition model is trained using at least two different types of target location signal light areas, the recognition model can comprehensively learn the features of at least two different types of target location signal light areas. By using the recognition model to process at least one target location signal light area, the status information of at least one corresponding target location signal light is acquired, thereby improving the accuracy of recognition of the status information of the target location signal light.

[0053] In some embodiments, see Figure 5 , at least one target location signal light area includes N location signal light areas corresponding to different types of signal lights, and when N is an integer greater than 1, in the above step S12, using the recognition model to process at least one target location signal light area to obtain the state information of at least one corresponding target location signal light may include the following steps:

[0054] S1211: Using the first preset network in the recognition model, extract features of the N signal light regions respectively to obtain the region features of the N signal light regions.

[0055] In some embodiments, see Figure 6 The recognition model may include a first preset network, a feature recombination network, and a second preset network. The first preset network may be used to obtain the features of the target part signal light area, or to obtain the state information of the target part signal light area. The feature recombination network is used to recombine the features of the first part and the second part. The second preset network is used to process the recombination features to obtain the final state information of the target part signal light area.

[0056] N position signal light areas can be input into the recognition model, and the first preset network of the recognition model can be used. The recognition model can include multiple parallel N first preset networks, so that feature extraction can be performed on the N position signal light areas in parallel and separately to obtain position features of the N position signal light areas. The position features can be high-dimensional features or high-level features, which can characterize the classification of the corresponding position signal light areas.

[0057] In some embodiments, N is 2, and the N signal light areas include a first signal light area and a second signal light area. The at least one target signal light area detected includes the first signal light area and the second signal light area, that is, includes the second signal light area, and the total number of target signal light areas included is greater than 1. For example, including the left brake light area and / or the right brake light area, and the high-mounted brake light area, the high-mounted brake light area and the left brake light area can be combined, and the high-mounted brake light area and the right brake light area can be combined to obtain the status information of the left brake light area and the right brake light area respectively.

[0058] In this embodiment, the first signal light area described in step S1211 refers to the left brake light area or the right brake light area.

[0059] In some embodiments, the first preset network (network 1) of the recognition model can be used to process the first part of the signal light area to obtain the corresponding first part feature. At the same time, the first preset network can be used to process the second part of the signal light area to obtain the second part feature. In this process, the same first preset network can be used, or the first preset networks can be used in parallel. The two parallel first preset networks perform feature extraction of the first part of the signal light area and the second part of the signal light area in parallel.

[0060] In some embodiments, the first part feature and the second part feature obtain weight parameters (or network parameters) of a shared first preset network. In addition, the first preset network for obtaining the first part feature and the second part feature can be regarded as the same network, so that the dimensions of the first part feature and the second part feature are the same.

[0061] In some embodiments, the first part feature includes a high-level feature extracted from the signal light area of ​​the first part, and the second part feature includes a high-level feature extracted from the signal light area of ​​the second part, or may be a feature output by the last layer before the first preset network outputs a classification result. The feature map obtained in the direction of reducing the size of the feature map is a high-level feature. The high-level feature pays more attention to the semantic information of the signal light area of ​​the target part and the global information of the signal light area of ​​the target part.

[0062] In some embodiments, the first preset network can be used to obtain the corresponding first part feature and the state information of the first part signal light area (classification result 1) for the first part signal light area. The first preset network can be used to obtain the second part feature and the state information of the second part signal light area (classification result 2) for the second part signal light area. Wherein, in the case of including the first part signal light area and the second part signal light area, the state information of the first part signal light area and the state information of the second part signal light area obtained by the first preset network in this step can be used as the reference state information of the corresponding signal light area, and can be used as the reference information of the final state information (classification result 3).

[0063] S1212: Reorganize the location features of the N location signal light areas to obtain reorganized features.

[0064] The feature recombination network of the recognition model can be used to recombine the part features of the N part signal light areas to obtain the recombined features. Specifically, the part features are sequentially subjected to convolution processing and attention processing to obtain the convolution features corresponding to the part features. Thus, based on the convolution features corresponding to the part features, the representation convolution features are obtained. Thus, the representation convolution features and at least part of the part features of the part features are used to perform attention processing to obtain the recombined features.

[0065] In some embodiments, one of the N part signal light areas can be used as the target part signal light area to be identified, and attention processing is performed using the representative convolutional features and the target part features to obtain a recombined feature; wherein the target part feature is obtained by extracting features of the target part signal light area using the first preset network. Thus, the recombined feature can be used for subsequent identification of the state information of the target part signal light area.

[0066] In some embodiments, N is 2, and the N part signal light areas include a first part signal light area and a second part signal light area. The characterizing convolution feature and the first part feature can be used to perform attention processing to obtain a recombined feature; wherein the first part feature is obtained by extracting features from the first part signal light area using a first preset network.

[0067] In some embodiments, the first part feature can be sequentially subjected to convolution processing (e.g., 1×1 convolution processing) and attention processing (Reshape processing) to obtain a first convolution feature. Also, the second part feature can be sequentially subjected to convolution processing (e.g., 1×1 convolution processing) and attention processing to obtain a second convolution feature. The dimensions of the first part feature and the second part feature are the same. This method can reduce the number of channels of the first part feature and the second part (e.g., reducing the number of channels to 1), thereby reducing the amount of computation of the network. Thus, based on the first convolution feature and the second convolution feature, for example, the first convolution feature and the second convolution feature can be matrix multiplied to obtain a representation convolution feature. The representation convolution feature can be expressed as a reorganized representation matrix. Furthermore, the representation convolution feature and the first part feature can be used to perform attention processing to obtain a reorganized feature.

[0068] In some embodiments, the step of performing recombination processing on the first part feature and the second part feature to obtain the recombined feature can be expressed as:

[0069] Feat′1=RS (C,M,N) (RS (C,M×N) (Feat1)·Rep_Mat)

[0070] Rep_Mat=RS (1,M×N) (Conv 1×1 (Feat1)) T RS (1,M×N) (Conv 1×1 (Feat2))

[0071] Among them, Feat′1 represents the reorganized feature, Rep_Mat represents the convolution feature, Feat1 represents the first part feature, and Feat2 represents the second part feature. RS represents attention processing (Reshape processing), Conv 1×1 represents 1×1 convolution processing, and “·” represents matrix multiplication.

[0072] The above-mentioned reorganization processing of the first part features and the second part draws on the idea of ​​the self-attention mechanism, calculates the correlation between the first part features after dimensionality reduction and the second part features after dimensionality reduction, and uses the weighted result of the second part features of the second part traffic light area to recombine the first part features of the first part traffic light area, so that the feature information of the second part traffic light area is integrated into the features of the first part traffic light area.

[0073] In addition, the above-mentioned feature reorganization network can be a lightweight feature reorganization module, and the dimension of the first part feature Feat1 (and the reorganized feature obtained after reorganization) before and after the feature reorganization process does not change. Therefore, as long as the size of the first part feature Feat1 remains unchanged, the above-mentioned feature reorganization network can also be replaced with any other network module, and this application does not limit this.

[0074] S1213: Process the reorganized features using the second preset network in the recognition model to obtain status information of the corresponding part signal light.

[0075] After obtaining the reorganization features corresponding to at least a partial signal light area, a target signal light area or a first signal light area, the reorganization features can be processed using the second preset network in the recognition model to obtain the corresponding status information of at least a partial signal light, a target signal light or a first signal light.

[0076] In some embodiments, N is 2, and the N signal light areas include a first signal light area and a second signal light area. The reorganized features can be processed using a second preset network to obtain the corresponding state information of the first signal light. If the input recognition model is the left brake light area and the high-mounted brake light area, the state information corresponding to the left brake light area is obtained. If the input recognition model is the right brake light area and the high-mounted brake light area, the state information corresponding to the right brake light area is obtained.

[0077] In some embodiments, if a combination of the left brake light area or the right brake light area and the high-mounted brake light area is input into a recognition model, status information of the corresponding left brake light area or the right brake light area and the high-mounted brake light area can be obtained.

[0078] In some embodiments, the second preset network may be composed of a convolutional layer and a fully connected layer, and the second preset network may adopt a structure such as a Transformer network, but the present application is not limited thereto.

[0079] In some embodiments, the first preset network may further include a classification layer, for example, the classification layer may be a fully connected layer. In the above step S1211, the classification layer may be used to process the location features of at least some of the location features of the N location signal light areas to obtain the state information corresponding to at least some of the location signal lights. Alternatively, the third preset network of the recognition model may be used to process at least some of the location features of the N location signal light areas to obtain the state information corresponding to at least some of the location signal lights.

[0080] In some embodiments, see Figure 7When at least one target part signal light area includes only one type of part signal light area, one type of part signal light area is used as the part signal light area to be processed. The above step S12, using the recognition model to process at least one target part signal light area to obtain the state information of at least one corresponding target part signal light, may include the following steps:

[0081] S1221: Using the first preset network in the recognition model to extract features of the signal light area of ​​the part to be processed, and obtaining features of the part to be processed in the signal light area of ​​the part to be processed; using the classification layer to process the features of the part to be processed, and obtaining status information of the signal light of the part to be processed.

[0082] When at least one target part signal light area includes only one type of part signal light area, the one type of part signal light area can be used as the part signal light area to be processed. When N is 2, when it is detected that only the first part signal light area or only the second part signal light area is included, the steps in this embodiment can be used to identify the status information.

[0083] In this step, the first preset network in the recognition model is used to extract features of the signal light area of ​​the to-be-processed part to obtain the features of the signal light area of ​​the to-be-processed part. The implementation process of the above embodiment can be referred to, and this application will not elaborate on it here.

[0084] In some embodiments, the first preset network may further include a classification layer, which may be a fully connected layer or other network with classification function, but the present application is not limited thereto. The classification layer may be used to process the features of the part to be processed to obtain the status information of the signal light of the part to be processed.

[0085] In some embodiments, the first preset network may not include a classification layer, and the classification layer and the first preset network may exist separately in the recognition model. The first preset network in the recognition model may be used to extract features of the signal light area of ​​the part to be processed to obtain features of the part to be processed in the signal light area of ​​the part to be processed; and then the classification layer in the recognition model may be used to process the features of the part to be processed to obtain status information of the signal light of the part to be processed.

[0086] S1231: Process the signal light area of ​​the part to be processed by using the third preset network of the recognition model to obtain the status information of the corresponding signal light of the part to be processed.

[0087] In some embodiments, the third preset network may have the same function as the first preset network. When the first preset network of the recognition model does not contain a classification layer, the third preset network may include the first preset network and the classification layer. If the first preset network includes a classification layer, the third preset network may be the same network as the first preset network. The third preset network of the recognition model may be used to process the signal light area of ​​the to-be-processed part to obtain the status information of the corresponding signal light of the to-be-processed part.

[0088] In some embodiments, when N is 2, when only the first signal light area or only the second signal light area is detected, refer to Figure 8 , when at least one target part signal light area only includes the first part signal light area, the first preset network or the third preset network (network 1) can be used to process the first part signal light area (for example, the left brake area) to obtain the corresponding state information of the first part signal light. The state information of the first part signal light includes a bright state and a non-bright state. The first preset network or the third preset network can be a lightweight classification network. In this process, the output high-level features (C, M, N), that is, the first part features, can be ignored, and the corresponding state information of the first part signal light can be directly obtained as the final state information.

[0089] See also Fig. 9 , when at least one target part signal light area only includes the second part signal light area, the second part signal light area is processed by the first preset network or the third preset network (network 1) to obtain the corresponding state information of the second part signal light. The first preset network or the third preset network can be a lightweight classification network. In this process, the output high-level features (C, M, N) that is, the second part features can be ignored, and the corresponding state information of the first part signal light can be directly used as the final state information.

[0090] In the present embodiment, since the recognition model is trained using at least two different types of target location signal light areas, that is, the recognition model learns the characteristics of the first location signal light area and the second location signal light area, wherein the first preset network or the third preset network of the recognition model also learns the characteristics of the first location signal light area and the second location signal light area, even if the first preset network of the recognition model is used to identify the status information, a more comprehensive judgment can be made, thereby improving the accuracy of identifying the status information of the target location signal light area.

[0091] In some implementations, for the above step S12, based on the hardware configuration, part of the network (the first preset network) or all the networks in the identification model may be selected to perform identification of the status information.

[0092] Among them, because the network of the recognition model is too complex, if the computing power of the hardware configuration of the device executing the above method cannot realize the recognition model, or the hardware configuration computing power of the device is limited, or there are requirements for the delay of the recognition model, etc., the first preset network of the recognition model can be used to obtain the status information of the signal light at the target part. If there is no restriction, the recognition model of the above embodiment is used to process and obtain the status information of the signal light at the target part.

[0093] See also Fig.10 , Fig.10 This is a flow chart of the second embodiment of the vehicle signal light recognition method of the present application. The method may include the following steps:

[0094] S21: Acquire a target image of the vehicle.

[0095] S22: Detecting the signal lights of the target part of the target image to obtain at least one signal light area of ​​the target part.

[0096] S23: Processing at least one target location signal light region using a recognition model to obtain state information of at least one corresponding target location signal light, wherein the recognition model is trained using at least two different types of target location signal light regions.

[0097] Wherein, at least one target location signal light area includes first location signal light areas and / or second location signal light areas of different types.

[0098] The first signal light area includes a left brake light area and / or a right brake light area, and the second signal light area includes a high-mounted brake light area.

[0099] In this embodiment, the specific implementation process of step S21 to step S23 can refer to the specific implementation process of step S11 to step S13 in the above embodiment, which will not be repeated here.

[0100] S24: Sending the status information of at least one corresponding target location signal light to the driving assistance system of the vehicle, so as to analyze the status information and the driving data of the vehicle and send prompt information.

[0101] The vehicle signal light recognition method of this embodiment can be applied to vehicle-mounted electronic equipment, which is described by taking this as an example.

[0102] After obtaining the status information of at least one target part signal light, the status information of at least one corresponding target part signal light can be sent to the vehicle's Advanced Driving Assistance System (ADAS), so that the driving assistance system uses the status information of at least one corresponding target part signal light and the vehicle's driving data for comprehensive analysis. If the analysis finds that there is a danger, a prompt message can be sent in time to remind the driver to avoid traffic accidents caused by driver distraction, resulting in property loss and personal injury.

[0103] For the above embodiment, it is taken as an example. Please refer to Fig.11 , Fig.11 1 is a flow chart of the third embodiment of the vehicle signal light recognition method of the present application. The method comprises the following steps:

[0104] S31: Acquire vehicle images in real time.

[0105] S32: Perform vehicle detection on the vehicle image to obtain a target image.

[0106] The target image includes a rear image of the vehicle.

[0107] S33: Perform brake light area detection on the target image to obtain at least one brake light image.

[0108] S34: Determine whether there is a hardware computing power limitation.

[0109] If the answer is yes, execute step S35.

[0110] If the judgment is no, execute step S36.

[0111] S35: Process at least one brake light image using a first preset network.

[0112] After executing this step, execute step S38.

[0113] S36: Detect whether there is a high-mounted brake light image in at least one brake light image.

[0114] If the answer is yes, execute step S37.

[0115] If the judgment is no, execute step S35.

[0116] S37: Process at least one brake light image using the recognition model.

[0117] The recognition model includes a first preset network, a feature recombination network and a second preset network.

[0118] S38: Obtain status information of at least one brake light.

[0119] S39: Output status information of at least one brake light.

[0120] For the above steps S31 to S39 of this embodiment of the present application, specific reference may be made to the specific implementation process of the above embodiment, and the present application will not elaborate on them here.

[0121] See also Fig.12 , Fig.12 The figure is a flow chart of an embodiment of a method for training a vehicle signal light recognition model of the present invention. The method may include the following steps:

[0122] S41: Acquire a sample image of a sample vehicle; wherein the sample image includes N part sample images of N part signal light areas corresponding to different types of signal lights, where N is an integer greater than 1.

[0123] The N part sample images corresponding to the N part signal light areas corresponding to the different types of signal lights of the sample vehicle can be combined to form a sample image of the sample vehicle, and input into the recognition model to be trained.

[0124] Among them, when N is 2, a first sample image including a first signal light area can be obtained, and the first signal light area can be a left brake light area or a right brake light area, and a second sample image including a second signal light area can be obtained, and the second signal light area can be a high-mounted brake light area.

[0125] The first sample image and the second sample image are combined to form a sample image of a sample vehicle, which is input into a recognition model to be trained.

[0126] S42: Using the recognition model to process the N part sample images, respectively obtain the part prediction state information corresponding to each of the N part signal light areas and the combined prediction state information, where the combined prediction state information is obtained based on the N part signal light areas.

[0127] The recognition model can be used to process N part sample images to obtain part prediction state information corresponding to each of the N part signal light areas, as well as combined prediction state information obtained based on corresponding part features of the N part signal light areas using the recognition model.

[0128] The recognition model includes a first preset network, a feature recombination network and a second preset network.

[0129] In some embodiments, the first preset network is used to extract features of N part signal light areas respectively, to obtain part features of the N part signal light areas and corresponding part prediction state information, where the part prediction state information is obtained by processing the part features using a classification layer.

[0130] In addition, when the first preset network includes a classification layer, that is, when it is the third preset network, multiple parallel third preset networks can be used to extract features of N position signal light areas respectively to obtain position features of the N position signal light areas and corresponding position prediction state information.

[0131] The feature recombination network is used to recombine the location features of the N location signal light areas to obtain the recombined features. The recombination process can refer to the implementation process of the above embodiment, and this application will not elaborate on it here.

[0132] The combined prediction state information may represent state information corresponding to one type of position signal lights in the N position signal light areas. The second preset network is used to process the reorganized features to obtain the combined prediction state information of one type of position signal lights in the corresponding N position signal lights.

[0133] In some embodiments, when N is 2, the first preset network (including a classification layer) or the third preset network of the recognition model is used to process the first sample image and the second sample image to obtain the predicted state information of the first signal light area and the predicted state information of the second signal light area, respectively.

[0134] The first sample image is processed by using the first preset network of the recognition model to obtain the corresponding first part feature; and the second sample image is processed by using the first preset network to obtain the second part feature. The first part feature and the second part feature are recombined to obtain the recombined feature; the recombined feature is processed by using the second preset network to obtain the corresponding first part signal light combined prediction state information.

[0135] The first part feature and the second part feature are recombined to obtain the recombined feature, including: sequentially performing convolution processing and attention processing on the first part feature to obtain the first convolution feature; and sequentially performing convolution processing and attention processing on the second part feature to obtain the second convolution feature. Based on the first convolution feature and the second convolution feature, a representation convolution feature is obtained; and the representation convolution feature and the first part feature are used to perform attention processing to obtain the recombined feature.

[0136] The implementation process of step S41 to step S42 in this embodiment can refer to the specific implementation process of the above embodiment, which will not be repeated here.

[0137] S43: Adjust the parameters of the recognition model based on the predicted state information of each part and the combined predicted state information to obtain a trained recognition model.

[0138] Based on the weights corresponding to the predicted state information of each part and the combined preset state information, the weights can be the same, or different weights can be set according to the importance and type of the part traffic lights, and based on the weights, the loss values ​​corresponding to the predicted state information of each part and the combined preset state information are obtained respectively. The parameters of the recognition model are adjusted using the loss values ​​corresponding to the predicted state information of each part and the combined preset state information, and the parameter adjustment includes the parameter adjustment of the first preset network or the third preset network, the feature recombination network, and the second preset network, so as to obtain a trained recognition model, so that the recognition model can learn the feature information of the N part traffic light areas.

[0139] When N is 2, during the network training of the recognition model, the high-mounted brake light area and the left brake light area or the right brake light area are input as a combination each time, and the predicted state information of the left brake light area and the right brake light area are predicted respectively. Therefore, the gradient of the predicted state information after combining the features of the high-mounted brake light area will be back-propagated to the first preset network, thereby guiding the learning of the first preset network, so that the prediction in training will be implicitly supervised by the features from the high-mounted brake light area. Even if the first preset network is used to obtain the state information of the target area, a more comprehensive judgment can be made compared to training the first preset network alone.

[0140] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0141] For the above embodiment, the present application also provides a vehicle signal light recognition device. Fig.13 , Fig.13 The first embodiment of the vehicle signal light recognition device of the present invention is shown in FIG.

[0142] The acquisition module 51 is used to acquire a target image of the vehicle.

[0143] The detection module 52 is used to detect the signal lights of the target part of the target image to obtain at least one signal light area of ​​the target part.

[0144] The recognition module 53 is used to process at least one target part signal light area using a recognition model to obtain state information of at least one corresponding target part signal light, wherein the recognition model is trained using at least two different types of target part signal light areas.

[0145] Wherein, at least one target part signal light area includes different types of first part signal light areas and / or second part signal light areas; the first part signal light area includes a left brake light area and / or a right brake light area, and the second part signal light area includes a high-mounted brake light area. The state information of at least one corresponding target part signal light includes a light state and a non-light state.

[0146] In some embodiments, the acquisition module 51 is used to acquire a target image of a vehicle, including: acquiring a vehicle image of the vehicle; acquiring a target image of the vehicle from the vehicle image, wherein the target image includes a rear image of the vehicle.

[0147] In some embodiments, at least one target part signal light area includes N part signal light areas corresponding to different types of signal lights, where N is an integer greater than 1. The recognition module 53 is used to process the at least one target part signal light area using the recognition model to obtain status information of at least one corresponding target part signal light, including: using the first preset network in the recognition model to extract features of the N part signal light areas respectively to obtain part features of the N part signal light areas; recombining the part features of the N part signal light areas to obtain recombined features; and using the second preset network in the recognition model to process the recombined features to obtain status information of the corresponding part signal light.

[0148] In some embodiments, the recognition module 53 is used to reorganize the part features of N part traffic light areas to obtain reorganized features, including: performing convolution processing and attention processing on each part feature in sequence to obtain the convolution features corresponding to each part feature; obtaining the representation convolution features based on the convolution features corresponding to each part feature; using the representation convolution features and at least part of the part features in each part feature to perform attention processing to obtain the reorganized features.

[0149] The first part of the signal light area is processed by the first preset network to obtain the corresponding first part feature; and the second part of the signal light area is processed by the first preset network to obtain the second part feature; the first part feature and the second part feature are recombined to obtain the recombined feature; the recombined feature is processed by the second preset network to obtain the corresponding first part signal light status information.

[0150] In some embodiments, N is 2, and the N part signal light areas include a first part signal light area and a second part signal light area. The recognition module 53 is used to perform attention processing using the representative convolution feature and at least part of the part features of each part feature to obtain a recombined feature, including: performing attention processing using the representative convolution feature and the first part feature to obtain a recombined feature; wherein the first part feature is obtained by extracting features of the first part signal light area using a first preset network.

[0151] In some embodiments, when at least one target part signal light area includes only one type of part signal light area, one type of part signal light area is used as the part signal light area to be processed. The recognition module 53 is used to extract features of the part signal light area to be processed using the first preset network in the recognition model to obtain the part features to be processed of the part signal light area to be processed; and to process the part features to be processed using the classification layer to obtain the state information of the part signal light to be processed. Alternatively, the part signal light area to be processed is processed using the third preset network of the recognition model to obtain the corresponding state information of the part signal light to be processed.

[0152] In some embodiments, the identification module 53 is further used to send the status information of at least one corresponding target location signal light to the driving assistance system of the vehicle, so as to use the status information and the driving data of the vehicle for analysis and send prompt information.

[0153] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0154] For the above embodiment, as an example, the present application also provides a vehicle signal light recognition device. Fig.14 , Fig.14 The second embodiment of the vehicle signal light recognition device of the present application is a schematic diagram of the structure of the vehicle signal light recognition device 60. The vehicle signal light recognition device 60 includes a processing terminal module 61, an image acquisition module 62, a vehicle detection module 63, a brake light area detection module 64, a brake light state recognition module 65 and an analysis and processing module 66.

[0155] The processing terminal module 61 is used to send a detection command to the receiving image acquisition module 62 to start detecting the signal light, and is used to receive the image information collected by the image acquisition module 62. The processing terminal module 61 is a hardware platform capable of performing a series of calculations, which can be a camera with good computing performance, or a hardware platform with computing capabilities such as a server deployed at the back end.

[0156] The image acquisition module 62 is used to continuously acquire the signal light area to be detected, acquire the vehicle image, and send it to the processing terminal module 61 for subsequent processing after receiving the detection command sent by the processing terminal module 61. In some application scenarios, the image acquisition module 62 can also be a camera device such as a surveillance camera. In some embodiments, in order to more accurately identify the signal light, the image acquisition module 62 needs to be able to completely capture the information of the vehicle's tail lights.

[0157] The vehicle detection module 63 is used to perform preliminary analysis on the vehicle image acquired by the image acquisition module 62, perform vehicle detection on the vehicle image, obtain a vehicle frame for each vehicle target, and assign an independent ID to the detected vehicle target for subsequent processing.

[0158] The brake light area detection module 64 is used to detect signal light areas (such as brake light areas) from the vehicle frame and mark the positions of different signal light areas to obtain at least one target area signal light area, that is, a signal light image of at least one target area signal light.

[0159] The brake light status recognition module 65 is used to select the recognition model or the first preset network for processing according to at least one detected target signal light area (such as the brake light area, which may include the number and type of brake lights), to obtain the status information of at least one corresponding target signal light area.

[0160] The analysis and processing module 66 is used to receive the status information of at least one corresponding target signal light area (such as the status information of the brake light) transmitted by the brake light status recognition module 65, and can be combined with other driving data of the vehicle for comprehensive analysis and processing. If the analysis shows that there is a potential danger, the driver will be reminded in time to avoid traffic accidents caused by the driver's distraction, resulting in property loss and personal injury.

[0161] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0162] For the above embodiment, the present application also provides a vehicle signal light recognition device. Fig.15 , Fig.15 The structure diagram of an embodiment of the training device for the recognition model of vehicle signal lights of the present application is shown in FIG. The training 70 of the recognition model of vehicle signal lights includes an acquisition module 71 , a recognition module 72 and an adjustment module 73 .

[0163] The acquisition module 71 is used to acquire sample images of sample vehicles; wherein the sample images include N part sample images of N part signal light areas corresponding to different types of signal lights, where N is an integer greater than 1.

[0164] The recognition module 72 processes the N part sample images using the recognition model to obtain the part prediction state information corresponding to each of the N part signal light areas and the combined prediction state information, where the combined prediction state information is obtained based on the N part signal light areas.

[0165] The recognition module 72 is used to adjust the parameters of the recognition model based on the predicted state information of each part and the combined predicted state information to obtain a trained recognition model.

[0166] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0167] For the above embodiment, the present application provides an electronic device, see Fig.16 , Fig.16 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 80 includes a memory 81 and a processor 82, wherein the memory 81 and the processor 82 are coupled to each other, the memory 81 stores program data, and the processor 82 is used to execute the program data to implement the steps in any of the embodiments of the above-mentioned vehicle signal light recognition method and the vehicle signal light recognition model training method.

[0168] In this embodiment, the processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 82 may also be any conventional processor, etc.

[0169] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0170] The method of the above embodiment can be implemented in the form of a computer program, so the present application proposes a storage device, see Fig.17 , Fig.17 The storage device 90 stores program data 91 that can be run by a processor, and the program data 91 can be executed by the processor to implement the steps of any of the above-mentioned vehicle signal light recognition method and vehicle signal light recognition model training method.

[0171] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0172] The storage device 90 of this embodiment can be a medium that can store program data 91, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the program data 91. The server can send the stored program data 91 to other devices for execution, or it can also execute the stored program data 91 by itself.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage device, which is a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of each implementation method of the present application.

[0177] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0178] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for identifying a vehicle signal light, characterized in that: The method comprises: Acquire a target image of a vehicle; Detecting a signal light at a target location on the target image to obtain at least one signal light region at the target location; Processing the at least one target part signal light region using a recognition model to obtain state information of at least one corresponding target part signal light, wherein the recognition model is trained using at least two different types of target part signal light regions; The at least one target location signal light area includes N location signal light areas corresponding to different types of signal lights, where N is an integer greater than 1; and the at least one target location signal light area is processed using a recognition model to obtain status information of at least one corresponding target location signal light, including: Using the first preset network in the recognition model, extracting features of the N signal light regions respectively to obtain the region features of the N signal light regions; Recombining the location features of the N location signal light regions to obtain recombined features; The reorganized features are processed using a second preset network in the recognition model to obtain status information of the corresponding part signal lights.

2. The method according to claim 1, characterized in that The reorganizing the position features of the N position signal light areas to obtain the reorganized features includes: Performing convolution processing and attention processing on each of the part features in sequence to obtain convolution features corresponding to each part feature; Based on the convolution features corresponding to the features of each part, a characterizing convolution feature is obtained; The characterizing convolutional features and at least some of the part features are used to perform attention processing to obtain the recombined features.

3. The method according to claim 2, characterized in that The N is 2, and the N signal light areas include a first signal light area and a second signal light area; The step of performing attention processing using the characterizing convolutional features and at least some of the part features to obtain the recombined features includes: The characterizing convolutional features and the first part features are used to perform attention processing to obtain the reorganized features; wherein the first part features are obtained by extracting features from the first part traffic light area using the first preset network.

4. The method according to claim 1, characterized in that When the at least one target part signal light area includes only one type of part signal light area, the one type of part signal light area is used as the part signal light area to be processed; The step of processing the at least one target location signal light region by using the recognition model to obtain status information of at least one corresponding target location signal light includes: Using the first preset network in the recognition model to extract features of the signal light area of ​​the part to be processed, to obtain features of the part to be processed of the signal light area of ​​the part to be processed; using the classification layer to process the features of the part to be processed, to obtain status information of the signal light of the part to be processed; or, The signal light area of ​​the to-be-processed part is processed using the third preset network of the recognition model to obtain the corresponding status information of the signal light of the to-be-processed part.

5. The method according to claim 1, characterized in that The at least one target location signal light area includes different types of left brake light areas or right brake light areas, and high-mounted brake light areas.

6. The method according to claim 1, characterized in that The method further comprises: The state information of the at least one corresponding target location signal light is sent to the driving assistance system of the vehicle, so as to analyze the state information and the driving data of the vehicle and send prompt information.

7. A method for training a vehicle signal light recognition model, characterized in that: The method comprises: Acquire a sample image of a sample vehicle; wherein the sample image includes N part sample images of N part signal light areas corresponding to different types of signal lights, where N is an integer greater than 1; Processing the N part sample images using a recognition model to obtain part prediction state information and combined prediction state information corresponding to the N part signal light areas, respectively, wherein the combined prediction state information is obtained based on the N part signal light areas; Adjusting the parameters of the recognition model based on the predicted state information of each part and the combined predicted state information to obtain a trained recognition model; The recognition model includes a first preset network, a feature recombination network, and a second preset network; wherein the first preset network is used to respectively extract features of the N position signal light areas to obtain position features and corresponding position prediction status information of the N position signal light areas, and the position prediction status information is obtained by processing the position features using a classification layer; the feature recombination network is used to recombine the position features of the N position signal light areas to obtain recombined features; the second preset network is used to process the recombined features to obtain combined prediction status information of a type of position signal lights among the corresponding N position signal lights.

8. The method according to claim 7, characterized in that The combined predicted state information indicates state information corresponding to one type of position signal light in the N position signal light areas.

9. The method according to claim 7, characterized in that: The adjusting the parameters of the recognition model based on the predicted state information of each part and the combined predicted state information to obtain a trained recognition model includes: Based on the weights corresponding to the predicted state information of each part and the combined preset state information, respectively obtain the loss values ​​corresponding to the predicted state information of each part and the combined preset state information; The parameters of the recognition model are adjusted using the predicted state information of each part and the loss value corresponding to the combined preset state information to obtain a trained recognition model.

10. An electronic device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement the steps of the method according to any one of claims 1 to 9.

11. A storage device, characterized in that: Program data that can be executed by a processor are stored, and the program data is used to implement the steps of the method according to any one of claims 1 to 9.

Images