Feature extraction method, device, vehicle orientation determination method, and vehicle-mounted terminal

Abstract

The embodiment of the present invention discloses a feature extraction method, device, vehicle orientation determination method, and vehicle-mounted terminal. The method includes: determining the width-to-height ratio of the vehicle to be detected according to the relative position relationship between the vehicle area and the license plate area; extracting a feature map of the vehicle area through a feature extraction layer in a vehicle structure line detection model; regressing the feature map according to the width-to-height ratio through a regression layer in the vehicle structure line detection model to determine the structure line information of the vehicle to be detected in the vehicle area; determining the orientation information corresponding to the structure line information according to the correspondence between the preset orientation information and the structure line information as the orientation information of the vehicle to be detected; wherein each feature vector of the feature map is obtained by superimposing an attention factor; the attention factor is determined according to the similarity between the feature vector and the global semantic features of all feature vectors in the semantic group. Applying the solution provided by the embodiment of the present invention can more accurately determine the orientation information of the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a feature extraction method and device, a vehicle orientation determination method, and a vehicle-mounted terminal. Background Art

[0002] With the development of science and technology, new concepts such as intelligent driving and driverless cars have emerged. High-precision vehicle detection, identification, tracking and estimation are important elements in road scene analysis and environmental perception, and are also an indispensable part of the field of intelligent driving. Estimation of the vehicle orientation of surrounding vehicles is an important part of perception technology. In addition to determining the vehicle orientation of surrounding vehicles in the field of intelligent driving, this application will also be found in other fields, such as road monitoring.

[0003] In the related art, a road image including the vehicle to be detected is usually collected by an image acquisition device, and pre-processed by denoising, smoothing filtering, and binarization threshold segmentation according to traditional image morphology to obtain a binary black-and-white image, and the vehicle edge line is detected by contour extraction and tracking, and the vehicle orientation information is determined according to the vehicle edge line. However, this method of determining the vehicle orientation is easily interfered by background information and cannot accurately extract the vehicle edge line, resulting in inaccurate determination of the vehicle orientation. Summary of the invention

[0004] The present invention provides a feature extraction method, device, vehicle orientation determination method, and vehicle-mounted terminal to more accurately determine vehicle orientation information. The specific technical solution is as follows.

[0005] In a first aspect, an embodiment of the present invention discloses a method for determining a vehicle orientation based on a structure line, comprising:

[0006] Acquire a road image containing a vehicle to be detected;

[0007] Determine a vehicle area and a license plate area where the vehicle to be detected is located in the road image;

[0008] Determining the width-to-height ratio of the vehicle to be detected according to the relative position relationship between the vehicle area and the license plate area;

[0009] A feature map of the vehicle area is extracted through a feature extraction layer in a vehicle structure line detection model; wherein the feature map includes a plurality of semantic groups; each feature vector in each semantic group is obtained by superimposing an attention factor; the attention factor is determined based on the similarity between the feature vector and the global semantic features of all feature vectors in the semantic group;

[0010] Regressing the feature map according to the aspect ratio through the regression layer in the vehicle structure line detection model to determine the structure line information of the vehicle to be detected in the vehicle area;

[0011] According to the preset correspondence between the orientation information and the structure line information, the orientation information corresponding to the structure line information is determined as the orientation information of the vehicle to be detected.

[0012] Optionally, the feature extraction layer includes: a convolution layer, an attention factor determination layer and an attention factor superposition layer; the step of extracting the feature map of the vehicle area through the feature extraction layer in the vehicle structure line detection model includes:

[0013] Extracting an initial feature map of the vehicle area through the convolution layer; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors;

[0014] By means of the attention factor determination layer, for each semantic group, a global semantic feature is determined according to all initial feature vectors in the semantic group, an initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature, and each initial attention factor is normalized to obtain an attention factor of each initial feature vector;

[0015] The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector through the attention factor superposition layer to obtain the corresponding feature vector, and the feature map containing each feature vector is used as the feature map of the vehicle area.

[0016] Optionally, the step of determining the global semantic feature according to all initial feature vectors in the semantic group includes:

[0017] According to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; wherein the x i is the i-th initial feature vector in the semantic group x, and m is the total number of all initial feature vectors in the semantic group x;

[0018] The step of determining the initial attention factor of each initial feature vector according to the similarity between each initial feature vector and the global semantic feature comprises:

[0019] According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where the c i is the initial attention factor of the i-th initial feature vector;

[0020] The step of normalizing each initial attention factor to obtain the attention factor of each initial feature vector includes:

[0021] Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; Said is the attention factor of the i-th initial feature vector, and ε is a constant;

[0022] The step of superimposing the attention factor of each initial feature vector to each initial feature vector to obtain each feature vector comprises:

[0023] According to the formula The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain the corresponding feature vector; where, The σ is a sigmoid function gate, the is the i-th feature vector, and the γ and β are the scaling coefficient and the shift coefficient corresponding to the semantic group obtained by training the vehicle structure line detection model.

[0024] Optionally, the vehicle structure line detection model is trained in the following manner:

[0025] Acquire multiple sample vehicle images and a sample width-height ratio of a sample vehicle in each sample vehicle image, as well as standard structural line information corresponding to each sample vehicle image;

[0026] Inputting the sample vehicle image into the vehicle structure line detection model;

[0027] Determine a sample feature vector of the sample vehicle image through the first model parameter of the convolution layer, and through the attention factor determination layer and the attention factor stacking layer;

[0028] Regressing the sample feature vector using the second model parameter of the regression layer and the sample width-height ratio corresponding to the sample vehicle image to obtain reference structure line information of the sample vehicle in the sample vehicle image;

[0029] Calculating the difference between the reference structure line information and the corresponding standard structure line information;

[0030] When the difference is greater than a preset difference threshold, adjusting the first model parameter and the second model parameter according to the difference, and returning to the step of inputting the sample vehicle image into the vehicle structure line detection model;

[0031] When the difference is less than the preset difference threshold, it is determined that the vehicle structure line detection model training is completed.

[0032] Optionally, the step of determining the aspect ratio of the vehicle to be detected according to the relative position relationship between the vehicle area and the license plate area includes:

[0033] Determining virtual height information of the vehicle to be detected according to the number of pixels in the height direction of the vehicle area;

[0034] Determine the number of two pixels between the license plate region boundary box and the vehicle region boundary box in the width direction;

[0035] Determining virtual width information of the vehicle to be detected according to the smaller number of pixels among the two numbers of pixels;

[0036] The aspect ratio of the vehicle to be detected is determined according to the virtual height information and the virtual width information.

[0037] Optionally, the correspondence between the structure line information and the orientation information is: a correspondence between a plurality of orientation categories and a structure line feature range; wherein each orientation category is a specific orientation selected from a set continuous orientation from a central point to a surrounding area;

[0038] The step of determining the orientation information corresponding to the structure line information according to the preset correspondence between the orientation information and the structure line information comprises:

[0039] A structural line feature range corresponding to the structural line information is determined, and orientation information corresponding to the structural line information is determined according to an orientation classification corresponding to the determined structural line feature range.

[0040] In a second aspect, an embodiment of the present invention discloses a feature extraction method in a deep learning network, comprising:

[0041] Obtain an image to be feature extracted;

[0042] According to a preset feature extraction rule, feature extraction is performed on the image to obtain an initial feature map; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors;

[0043] For each semantic group, a global semantic feature is determined according to all initial feature vectors in the semantic group, and an initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature; each initial attention factor is normalized to obtain an attention factor of each initial feature vector;

[0044] The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain the corresponding feature vector; and the feature map containing each feature vector is used as the feature map of the image.

[0045] Optionally, for each semantic group, determining a global semantic feature according to all initial feature vectors in the semantic group, determining an initial attention factor of each initial feature vector according to a similarity between each initial feature vector and the global semantic feature; and normalizing each initial attention factor to obtain the attention factor of each initial feature vector comprises:

[0046] For each semantic group, according to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; wherein the x i is the i-th initial feature vector in the semantic group x, and m is the total number of all initial feature vectors in the semantic group x;

[0047] According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where the c i is the initial attention factor of the i-th initial feature vector;

[0048] Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; Said is the attention factor of the i-th initial feature vector, and ε is a constant.

[0049] Optionally, the step of superimposing the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector includes:

[0050] According to the formula The attention factor of each initial feature vector is added to the corresponding initial feature vector to obtain the corresponding feature vector;

[0051] in, The σ is a sigmoid function gate, the is the i-th initial eigenvector x i The attention factor, is the i-th feature vector, and the γ and β are the scaling coefficient and shift coefficient corresponding to the semantic group obtained by training the deep learning network.

[0052] In a fourth aspect, an embodiment of the present invention discloses a feature extraction device in a deep learning network, comprising:

[0053] An acquisition module is configured to acquire an image to be feature extracted;

[0054] An extraction module is configured to perform feature extraction on the image according to a preset feature extraction rule to obtain an initial feature map; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors;

[0055] A determination module is configured to determine, for each semantic group, a global semantic feature according to all initial feature vectors in the semantic group, determine an initial attention factor of each initial feature vector according to a similarity between each initial feature vector and the global semantic feature; normalize each initial attention factor to obtain an attention factor of each initial feature vector;

[0056] The superposition module is configured to superimpose the attention factor of each initial feature vector onto the corresponding initial feature vector to obtain the corresponding feature vector; and use the feature map containing each feature vector as the feature map of the image.

[0057] In a fourth aspect, an embodiment of the present invention discloses a vehicle-mounted terminal, comprising: a processor and a camera device; the processor comprises: an image acquisition module, a region determination module, a ratio determination module, a feature extraction module, a structure line determination module and a direction determination module;

[0058] The image acquisition module is used to acquire a road image containing a vehicle to be detected;

[0059] The area determination module is used to determine the vehicle area and license plate area where the vehicle to be detected is located in the road image;

[0060] The ratio determination module is used to determine the width-to-height ratio of the vehicle to be detected according to the relative position relationship between the vehicle area and the license plate area;

[0061] The feature extraction module is used to extract a feature map of the vehicle area through a feature extraction layer in a vehicle structure line detection model; wherein the feature map includes a plurality of semantic groups; each feature vector in each semantic group is obtained by superimposing an attention factor; the attention factor is determined based on the similarity between the feature vector and the global semantic features of all feature vectors in the semantic group;

[0062] The structure line determination module is used to regress the feature map according to the aspect ratio through the regression layer in the vehicle structure line detection model to determine the structure line information of the vehicle to be detected in the vehicle area;

[0063] The orientation determination module is used to determine the orientation information corresponding to the structure line information according to the preset correspondence between the orientation information and the structure line information, as the orientation information of the vehicle to be detected.

[0064] Optionally, the feature extraction layer includes: a convolution layer, an attention factor determination layer and an attention factor superposition layer; the feature extraction module, when extracting the feature map of the vehicle area through the feature extraction layer in the vehicle structure line detection model, includes:

[0065] Extracting an initial feature map of the vehicle area through the convolution layer; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors;

[0066] By means of the attention factor determination layer, for each semantic group, a global semantic feature is determined according to all initial feature vectors in the semantic group, an initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature, and each initial attention factor is normalized to obtain an attention factor of each initial feature vector;

[0067] The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector through the attention factor superposition layer to obtain the corresponding feature vector, and the feature map containing each feature vector is used as the feature map of the vehicle area.

[0068] Optionally, the feature extraction module, when determining the global semantic feature according to all the initial feature vectors in the semantic group, includes:

[0069] According to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; wherein the x i is the i-th initial feature vector in the semantic group x, and m is the total number of all initial feature vectors in the semantic group x;

[0070] The feature extraction module, when determining the initial attention factor of each initial feature vector according to the similarity between each initial feature vector and the global semantic feature, comprises:

[0071] According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where the c i is the initial attention factor of the i-th initial feature vector;

[0072] The feature extraction module normalizes each initial attention factor to obtain the attention factor of each initial feature vector, including:

[0073] Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; Said is the attention factor of the i-th initial feature vector, and ε is a constant;

[0074] The feature extraction module, when superimposing the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector, includes:

[0075] According to the formula The attention factor of each initial feature vector is superimposed on each initial feature vector to obtain the corresponding feature vector; where, The σ is a sigmoid function gate, the is the i-th feature vector, and the γ and β are the scaling coefficient and the shift coefficient corresponding to the semantic group obtained by training the vehicle structure line detection model.

[0076] Optionally, the processor further includes: a model training module, configured to train the vehicle structure line detection model by performing the following operations:

[0077] Acquire multiple sample vehicle images and a sample width-height ratio of a sample vehicle in each sample vehicle image, as well as standard structural line information corresponding to each sample vehicle image;

[0078] Inputting the sample vehicle image into the vehicle structure line detection model;

[0079] Determine a sample feature vector of the sample vehicle image through the first model parameter of the convolution layer, and through the attention factor determination layer and the attention factor stacking layer;

[0080] Regressing the sample feature vector using the second model parameter of the regression layer and the sample width-height ratio corresponding to the sample vehicle image to obtain reference structure line information of the sample vehicle in the sample vehicle image;

[0081] Calculating the difference between the reference structure line information and the corresponding standard structure line information;

[0082] When the difference is greater than a preset difference threshold, adjusting the first model parameter and the second model parameter according to the difference, and returning to the operation of inputting the sample vehicle image into the vehicle structure line detection model;

[0083] When the difference is less than the preset difference threshold, it is determined that the vehicle structure line detection model training is completed.

[0084] Optionally, the ratio determination module is specifically used to:

[0085] Determining virtual height information of the vehicle to be detected according to the number of pixels in the height direction of the vehicle area;

[0086] Determine the number of two pixels between the license plate region boundary box and the vehicle region boundary box in the width direction;

[0087] Determining virtual width information of the vehicle to be detected according to the smaller number of pixels among the two numbers of pixels;

[0088] The aspect ratio of the vehicle to be detected is determined according to the virtual height information and the virtual width information.

[0089] Optionally, the correspondence between the structure line information and the orientation information is: a correspondence between a plurality of orientation categories and a structure line feature range; wherein each orientation category is a specific orientation selected from a set continuous orientation from a central point to a surrounding area;

[0090] The orientation determination module is specifically used for:

[0091] A structural line feature range corresponding to the structural line information is determined, and orientation information corresponding to the structural line information is determined according to an orientation classification corresponding to the determined structural line feature range.

[0092] From the above content, it can be seen that the feature extraction method, device, vehicle orientation determination method, and vehicle-mounted terminal provided by the embodiments of the present invention superimpose an attention factor in each feature vector when extracting image features, and use the attention factor to adjust the importance of the feature vector, which can enhance the detection characteristics of the model and suppress the influence of similar patterns and noisy backgrounds in the image; when regressing the feature map in the regression layer to determine the structural line information of the vehicle, the width and height ratio of the vehicle is used, so that the structural line of the vehicle can be regressed more accurately. When the determined vehicle structural line is more accurate, the determined vehicle orientation is also more accurate. Of course, it is not necessary to achieve all of the advantages described above at the same time for any product or method implementing the present invention.

[0093] The innovative features of the embodiments of the present invention include:

[0094] 1. The attention factor and the aspect ratio introduced in the embodiment of the present invention can respectively enhance the importance of the feature vector in the feature extraction stage and improve the accuracy of the regression stage, thereby improving the accuracy of the vehicle orientation.

[0095] 2. When determining the attention factor, the global semantic feature is determined based on all the feature vectors in the semantic group, and then the initial attention factor of each feature vector is determined based on the similarity between each feature vector and the global semantic feature, and the initial attention factor is normalized; the above operations provide an easier to implement solution for adding attention factors.

[0096] 3. Convert the continuous orientation into discontinuous orientation, that is, divide it into multiple orientation categories, and determine which orientation category the vehicle belongs to based on the structural line information, which can simplify the vehicle orientation processing process and improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0098] Figure 1 A schematic flow chart of a method for determining a vehicle orientation based on structure lines provided in an embodiment of the present invention;

[0099] Figure 2 A reference diagram of the relative position relationship between the vehicle area and the license plate area provided in an embodiment of the present invention;

[0100] Figure 3 A schematic diagram of a vehicle structure line provided by an embodiment of the present invention;

[0101] Figure 4 A schematic diagram of a structure of a vehicle structure line detection model provided by an embodiment of the present invention;

[0102] Figure 5 A reference schematic diagram of superimposed attention factors provided by an embodiment of the present invention;

[0103] Figure 6 A schematic diagram of 8-direction classification provided by an embodiment of the present invention;

[0104] Figure 7 A schematic diagram of the structure of a vehicle-mounted terminal provided by an embodiment of the present invention;

[0105] Figure 8 A schematic diagram of a flow chart of a feature extraction method in a deep learning network provided by an embodiment of the present invention;

[0106] Fig. 9 A schematic diagram of the structure of a feature extraction device in a deep learning network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0107] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0108] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0109] The embodiment of the present invention discloses a feature extraction method, device, vehicle orientation determination method, and vehicle-mounted terminal, which can more accurately determine the orientation information of the vehicle. The embodiment of the present invention is described in detail below.

[0110] Figure 1 A flow chart of a method for determining a vehicle orientation based on a structure line provided in an embodiment of the present invention. The method is applied to an electronic device. The electronic device may be a common computer, a server, a mobile terminal, a vehicle-mounted terminal, etc. The method specifically includes the following steps.

[0111] S110: Acquire a road image containing a vehicle to be detected.

[0112] Among them, the road image includes not only the vehicle to be detected, but also various backgrounds, such as traffic signs, street light poles, trees, buildings, pedestrians and other backgrounds.

[0113] The road image may include but is not limited to an image captured by an image acquisition device in the current vehicle. The image acquisition device may be located near the inside of the front windshield of the current vehicle, or near the inside of the rear windshield. The road image may also be an image captured in real time by the image acquisition device, so that the orientation of the vehicles to be detected around the current vehicle can be determined in real time through this embodiment.

[0114] In this embodiment, in order to determine the orientation information of the vehicle to be detected, the following steps may be performed. The orientation of the vehicle to be detected may be the orientation of the vehicle to be detected relative to the current vehicle, or relative to the image acquisition device, or relative to a selected point.

[0115] S120: Determine the vehicle area and license plate area where the vehicle to be detected is located in the road image.

[0116] The vehicle region may be a region including all pixels of the vehicle to be detected, and the license plate region may be a region including all pixels of the license plate of the vehicle to be detected. The vehicle region and the license plate region are in one-to-one correspondence, and the license plate region is located inside the vehicle region. Both the vehicle region and the license plate region may be represented by regions surrounded by rectangular frames, and specifically may be represented by the coordinates of the diagonal points of the rectangle.

[0117] In one implementation of this step, a vehicle detection model may be used to determine the vehicle area and license plate area where the vehicle to be detected is located in the road image. The vehicle detection model is used to determine the vehicle area and license plate area where the vehicle to be detected is located in the road image based on pre-trained model parameters. The vehicle detection model may be pre-trained based on a large number of sample vehicle images and annotated vehicle areas and license plate areas. The vehicle detection model may be trained using a convolutional neural network. Due to the local connection and weight sharing characteristics of the convolutional layer in the convolutional neural network, the model parameters that need to be trained are greatly reduced, thereby simplifying the model training process and improving the training efficiency.

[0118] In another embodiment, the number of pixels in the vehicle area can be set to determine the vehicle area and the corresponding license plate area whose total number of pixels is greater than a preset threshold from the road image. In this way, the vehicles to be detected that are relatively close to the current vehicle can be screened out from the road image, and the orientation of the vehicles to be detected can be determined.

[0119] See also Figure 2 As shown in FIG. 1 , a reference image of the vehicle area and the license plate area in the road image is shown. The area in the large black frame is the vehicle area, and the area in the small black frame is the license plate area.

[0120] S130: Determine the width-to-height ratio of the vehicle to be detected according to the relative position relationship between the vehicle area and the license plate area.

[0121] The aspect ratio may be understood as the ratio between the width and height of the vehicle to be detected.

[0122] This step may specifically include: determining the relative position relationship between the vehicle area and the license plate area, and determining the width-to-height ratio of the vehicle to be detected based on the determined relative position relationship. Determining the width-to-height ratio of the vehicle to be detected using the relative position relationship between the vehicle area and the license plate area can determine the width-to-height ratio of the vehicle relatively quickly and accurately.

[0123] The relative position relationship between the vehicle area and the license plate area may be the relative position relationship between the center point of the vehicle area and the center point of the license plate area, or may be the relative position relationship between the bounding box of the vehicle area and the bounding box of the license plate area.

[0124] When determining the aspect ratio of the vehicle to be detected based on the relative position relationship, it can specifically include: judging whether the license plate area is located at the center of the vehicle area; if so, directly determining the aspect ratio of the vehicle to be detected based on the width and height of the vehicle area; if not, determining the aspect ratio of the vehicle to be detected based on the relative distance between the bounding box of the license plate area and the bounding box of the vehicle area.

[0125] When determining whether the license plate area is located at the center of the vehicle area, it can be determined whether the center point of the vehicle area coincides with the center point of the license plate area. If so, it is determined that the license plate area is located at the center of the vehicle area.

[0126] S140: extracting a feature map of the vehicle area through a feature extraction layer in the vehicle structure line detection model.

[0127] Among them, the feature map contains multiple semantic groups; each feature vector in each semantic group is obtained by superimposing an attention factor. The attention factor can also be understood as the importance coefficient of the feature vector in the semantic group. The attention factor is determined based on the similarity between the feature vector and the global semantic features of all feature vectors in the semantic group. The more similar the feature vector is to the global semantic features, the larger the attention factor of the feature vector is, and the greater the role of the feature vector superimposed with the attention factor in the feature map will be. Each feature vector corresponds to a spatial position in the vehicle area. Superimposing an attention factor for each spatial position can adjust the importance of each feature vector, so that each semantic group can autonomously enhance its learning expression and suppress possible noise.

[0128] The feature map can contain multiple channels. For each channel, the channel can be divided into G semantic groups, where G is preset. The multiple semantic groups of each channel do not affect each other, and the attention factor is determined based on the feature vector and global semantic features within each semantic group. Each semantic group corresponds to a specific type of entity. For example, the lights, tires, windows, license plates, doors, etc. in a vehicle can correspond to different semantic groups.

[0129] Since noise and similar objects inevitably exist in images, in order to obtain good feature separation, the overall information of the group space can be used. The global features of the entire space are not dominated by noise, so the learning of semantic features of key areas can be further enhanced.

[0130] In this embodiment, the vehicle structure line detection model can be a model trained by using a convolutional neural network. The vehicle structure line detection model is used to determine the structure line information of the vehicle to be detected from the vehicle area according to the pre-trained model parameters and the width-height ratio of the vehicle to be detected. The vehicle structure line detection model can be obtained by pre-training the neural network model using a large number of sample vehicle images and annotated standard structure line information.

[0131] The vehicle structure line detection model may include a feature extraction layer and a regression layer. The step of extracting a feature map of a vehicle region through the feature extraction layer in the vehicle structure line detection model may include: extracting a feature vector of the vehicle region through a first model parameter of the feature extraction layer. The first model parameter is a model parameter of the vehicle structure line detection model, which is obtained by training samples.

[0132] The feature extraction layer may include multiple convolutional layers for extracting image features.

[0133] S150: Through the regression layer in the vehicle structure line detection model, the feature map is regressed according to the aspect ratio to determine the structure line information of the vehicle to be detected in the vehicle area.

[0134] The structural lines of the vehicle may include the dividing lines between different planes of the vehicle and the boundary lines of the vehicle. The structural line information may be understood as the graphic information composed of the structural lines of the vehicle.

[0135] For example, Figure 3 The figure shows a schematic diagram of a vehicle structure line. The vehicle structure line is represented by a black or white line. There is a certain angle and length ratio between the various structure lines of the vehicle, and the structure line information may include information such as the angle and length ratio between the structure lines.

[0136] This step may specifically include: regressing the feature map through the second model parameter of the regression layer and the aspect ratio to obtain the structural line information of the vehicle to be detected.

[0137] The second model parameters are model parameters of the vehicle structure line detection model, which are obtained by training samples.

[0138] The width-to-height ratio of a vehicle can characterize the structure between different planes of the vehicle. According to the width-to-height ratio, the rear plane and side plane of the vehicle can be distinguished more accurately, and the structural line between the rear plane and the side plane can be distinguished more accurately.

[0139] S160: According to the preset correspondence between the orientation information and the structure line information, the orientation information corresponding to the structure line information is determined as the orientation information of the vehicle to be detected.

[0140] The correspondence between the orientation information and the structure line information can be determined by analyzing a large number of sample vehicle images and corresponding standard orientation information in advance.

[0141] From the above content, it can be seen that in this embodiment, an attention factor is superimposed in each feature vector when extracting image features. The attention factor is used to adjust the importance of the feature vector, which can enhance the features that need to be detected by the model and suppress the influence of similar patterns and noisy background in the image. When the feature map is regressed in the regression layer to determine the structural line information of the vehicle, the width and height ratio of the vehicle is used, so that the structural line of the vehicle can be regressed more accurately. When the determined vehicle structural line is more accurate, the determined vehicle orientation is also more accurate.

[0142] At the same time, the attention factor is only guided by the similarity between the global features and local feature descriptions within each semantic group, and its calculation is very lightweight and has a small amount of computation, which improves the processing efficiency of the entire process.

[0143] In another embodiment of the present invention, based on Figure 1 In the embodiment shown, the feature extraction layer may include: a convolution layer, an attention factor determination layer and an attention factor stacking layer. Figure 4 As shown, it includes a convolution layer, an attention factor determination layer, an attention factor superposition layer and a regression layer. In step S140, the step of extracting a feature map of the vehicle area through the feature extraction layer in the vehicle structure line detection model includes the following steps 1a to 3a.

[0144] Step 1a: Through the convolutional layer, extract the initial feature map of the vehicle area.

[0145] The initial feature map includes multiple semantic groups, and each semantic group includes multiple initial feature vectors. The convolution layer inputs the extracted initial feature map into the attention factor determination layer. The initial feature vector is a feature vector without superimposed attention factors.

[0146] In this embodiment, there may be multiple convolutional layers. The input of the first convolutional layer is the vehicle area, and the output of the last convolutional layer is the initial feature map. The input of the middle convolutional layer is the output of the previous convolutional layer, and the output of the middle convolutional layer is the input of the next convolutional layer.

[0147] Step 2a: Through the attention factor determination layer, for each semantic group, the global semantic feature is determined based on all the initial feature vectors in the semantic group, the initial attention factor of each initial feature vector is determined based on the similarity between each initial feature vector and the global semantic feature, and each initial attention factor is normalized to obtain the attention factor of each initial feature vector.

[0148] The attention factor determination layer receives the feature map input by the convolution layer and performs the operations of determining the global semantic features, determining the initial attention factor, normalization and attention factor in step 2. The attention factor determination layer inputs the determined attention factor to the attention factor superposition layer.

[0149] Step 3a: Through the attention factor superposition layer, the attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain the corresponding feature vector, and the feature map containing each feature vector is used as the feature map of the vehicle area.

[0150] In summary, in this embodiment, when determining the attention factor, the global semantic feature is determined based on all feature vectors in the semantic group, and then the initial attention factor of each feature vector is determined based on the similarity between each feature vector and the global semantic feature, and the initial attention factor is normalized; the above operations provide an easier to implement solution for adding attention factors.

[0151] In another embodiment of the present invention, based on the above embodiment, the step of determining the global semantic feature according to all the initial feature vectors in the semantic group in step 2a may include:

[0152] According to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g.

[0153] Among them, x i is the i-th initial feature vector in the semantic group χ, and m is the total number of all initial feature vectors in the semantic group χ. There are m initial feature vectors in the semantic group χ. gp It can also be called a spatial average function.

[0154] The step of determining the initial attention factor of each initial feature vector according to the similarity between each initial feature vector and the global semantic feature in step 2a includes:

[0155] According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor for each initial feature vector.

[0156] Among them, c i is the initial attention factor of the i-th initial feature vector. In this embodiment, the dot product between the initial feature vector and the global semantic feature is used to calculate the similarity between the two. i =g·x i You can also use c i =||g||||x i ||cosθ i To express, θ i is g and xi The angle between || || is the modulus symbol. From the expression ||g||||x i ||cosθ i It can be seen that when x i The closer the direction is to g, the i The larger the length of x i The larger the initial attention factor, the better.

[0157] The step of normalizing each initial attention factor in step 2a to obtain the attention factor of each initial feature vector includes:

[0158] Using formula Each initial attention factor is normalized to obtain the attention factor of each initial feature vector.

[0159] in, is the attention factor of the i-th initial eigenvector, ε is a constant added for numerical stability.

[0160] In space, c i Normalization can prevent the distribution bias of attention factors between different input samples in the same semantic group.

[0161] In step 3a, the step of superimposing the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector includes:

[0162] According to the formula The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain each feature vector.

[0163] in, σ is the sigmoid function gate, is the i-th feature vector, γ and β are the scaling coefficients and shift coefficients corresponding to the semantic group obtained when training the vehicle structure line detection model, each semantic group corresponds to a set of γ and β, and different semantic groups correspond to different γ and β. Specifically, γ and β can be determined by the gradient descent method when training the vehicle structure line detection model.

[0164] In this embodiment, c i After normalization, c i The scale information is lost. In order to restore the scale information, the scaling factor and shift coefficients γ and β are introduced, and then Transform into a i , so that a i With scale information, enhancement.

[0165] The expression of the sigmoid function can be σ(x)=1 / (1+e -x ), x is the amount of the input function. This function is the core function of the attention mechanism. This function makes the output between 0-1, 1 represents strong attention, and 0 represents no attention. This function plays a role in regulating attention.

[0166] See also Figure 5 , input the vehicle area into the convolution layer, and the initial feature map obtained contains different channels, C is one of the channels, m = H × W, H and W are the height and width of the image respectively. For each semantic group in the initial feature map of this channel, g can be calculated separately, c can be obtained by dot multiplication, normalized N, a is calculated, and the σ function is input and compared with x i The feature map is obtained by performing operations such as dot multiplication as the output of the feature extraction layer. The feature map is input to the regression layer. The above process converts the semantic group χ in the initial feature map into

[0167] In summary, this embodiment adds an attention factor to the initial feature map, which can improve the spatial distribution of different feature vectors within the semantic group, which can statistically produce a larger variance, strengthen the feature learning of the semantic area, and compress noise and interference.

[0168] In another embodiment of the present invention, based on Figure 1 In the illustrated embodiment, the following steps 1b to 6b are used to train the vehicle structure line detection model:

[0169] Step 1b: Obtain multiple sample vehicle images and the sample width-to-height ratio of the sample vehicle in each sample vehicle image, as well as standard structural line information corresponding to each sample vehicle image.

[0170] Step 2b: Input the sample vehicle image into the vehicle structure line detection model.

[0171] Step 3b: Determine a sample feature vector of the sample vehicle image through the first model parameters of the convolutional layer, and through the attention factor determination layer and the attention factor stacking layer.

[0172] The sample feature vector is obtained by superimposing the attention factor, and the attention factor is determined according to the similarity between the sample feature vector and the global semantic features of all feature vectors in the semantic group.

[0173] The specific implementation of this step can also refer to the implementation in the embodiment shown in steps 1a to 3a.

[0174] In this step, the initial values ​​of the scaling coefficients and shift coefficients γ and β can be set, and the sample feature vector of the sample vehicle image can be determined through the first model parameter of the convolution layer, and according to the values ​​of γ and β, through the attention factor determination layer and the attention factor superposition layer.

[0175] Step 4b: Regress the sample feature vector through the second model parameter of the regression layer and the sample width-height ratio corresponding to the sample vehicle image to obtain the reference structure line information of the sample vehicle in the sample vehicle image.

[0176] Step 5b: Calculate the difference between the reference structure line information and the corresponding standard structure line information.

[0177] Specifically, a loss function may be used to calculate the difference between the reference structure line information and the corresponding standard structure line information.

[0178] Step 6b: When the difference is greater than a preset difference threshold, the first model parameter and the second model parameter are adjusted according to the difference, and the step of inputting the sample vehicle image into the vehicle structure line detection model in step 2b is returned to execution. When the difference is less than the preset difference threshold, it is determined that the vehicle structure line detection model training is completed.

[0179] When the difference is equal to a preset difference threshold, the first model parameter and the second model parameter can be adjusted according to the difference, and it can also be determined that the vehicle structure line detection model training is completed.

[0180] The initial values ​​of the first model parameter and the second model parameter may be preset. The reference structure information determined according to the first model parameter and the second model parameter may have certain differences with the standard structure line information, and the first model parameter and the second model parameter may be continuously corrected according to the difference until the reference structure information is very similar to the standard structure line information.

[0181] In another embodiment, when the difference is greater than a preset difference threshold, the values ​​of γ and β and the first model parameter and the second model parameter are adjusted according to the difference, and the step of inputting the sample vehicle image into the vehicle structure line detection model in step 2b is returned to execution. When the difference is less than the preset difference threshold, the current values ​​of γ and β and the current values ​​of the first model parameter and the second model parameter are trained. When adjusting the values ​​of γ and β according to the difference, a gradient descent method can be used.

[0182] In order to make the model training converge quickly, a neural network model that has been trained in the field of image detection can be selected as a vehicle structure line detection model. On the basis of the vehicle structure line detection model, the model parameters are continuously adjusted until the model training is completed.

[0183] In summary, this embodiment provides a specific implementation method for training a vehicle structure line detection model. By using a large number of sample vehicle images and sample width-height ratios and corresponding standard structure line information, and continuously adjusting model parameters, the trained model can be more accurate in detecting structure lines.

[0184] In another embodiment of the present invention, based on Figure 1 In the illustrated embodiment, the step of determining the width-to-height ratio of the vehicle to be detected according to the relative positional relationship between the vehicle area and the license plate area in step S130 may include the following steps 1c to 4c.

[0185] Step 1c: Determine the virtual height information of the vehicle to be detected according to the number of pixels in the vehicle area in the height direction. Specifically, the number of pixels in the vehicle area in the height direction can be directly determined as the virtual height information of the vehicle to be detected.

[0186] Step 2c: Determine the number of two pixels between the license plate region bounding box and the vehicle region bounding box in the width direction.

[0187] The license plate area is located inside the vehicle area. The number of pixels between the left bounding box of the license plate area and the left bounding box of the vehicle area can be calculated, and the number of pixels between the right bounding box of the license plate area and the right bounding box of the vehicle area can be calculated. When calculating the number of pixels between bounding boxes, the number of pixels on the vertical line between the bounding boxes can be used as the number of pixels between the bounding boxes.

[0188] The two pixel numbers include the number of pixels between the left bounding boxes and the number of pixels between the right bounding boxes, respectively.

[0189] Step 3c: Determine the virtual width information of the vehicle to be detected according to the smaller number of pixels between the two numbers of pixels. Specifically, the smaller number of pixels between the two numbers of pixels can be directly determined as the virtual width information of the vehicle to be detected.

[0190] Step 4c: Determine the aspect ratio of the vehicle to be detected according to the virtual height information and the virtual width information.

[0191] The ratio between the virtual height information and the virtual width information, or the ratio between the virtual width information and the virtual height information, may be determined as the width-to-height ratio of the vehicle to be detected.

[0192] In summary, this embodiment provides an implementation method for determining the aspect ratio of a vehicle to be detected. On the basis of meeting certain accuracy requirements, this implementation method can determine the aspect ratio more efficiently.

[0193] In another embodiment of the present invention, based on Figure 1In the illustrated embodiment, the correspondence between the structure line information and the orientation information is as follows: a correspondence between multiple orientation classifications and structure line feature ranges. Each orientation classification is a specific orientation selected from a set center point to a continuous orientation of a surrounding area. The center point may also be the optical center of the camera, or may be another pre-set point. Each orientation classification may correspond to an orientation range, which includes multiple structure line features, and the multiple structure line features constitute a corresponding structure line feature range.

[0194] For example, the surrounding area of ​​a set center point can be set to 8 orientation categories. Figure 6 This is a schematic diagram of the 8-direction classification. Taking the camera in car A as the center point, the surrounding 360-degree area can be divided into 8 directions, where each direction classification is represented by the direction of a black line. The direction of car B belongs to the same direction classification as its direction.

[0195] Step S160, determining the orientation information corresponding to the structure line information according to the preset correspondence between the orientation information and the structure line information, comprises:

[0196] A structural line feature range corresponding to the structural line information is determined, and orientation information corresponding to the structural line information is determined according to an orientation classification corresponding to the determined structural line feature range.

[0197] The structural line feature range may include a direction vector interval corresponding to the structural line information.

[0198] In one embodiment, a classification model based on multiple orientation classifications may be pre-trained, and the structural line information may be input into the classification model. The classification model may determine the orientation classification corresponding to the input structural line information based on the correspondence between the multiple orientation classifications obtained through training and the structural line feature ranges. When training the classification model, a large number of pre-acquired sample vehicle images and corresponding annotated vehicle orientations may be used to train the neural network model.

[0199] When dividing the orientation categories, we can start from 0 degrees, and divide the orientation categories into 45 degree intervals, for a total of 8 categories. When annotating the vehicle orientation data, if the vehicle orientation in the sample vehicle image does not fall exactly into one of the 8 categories, then select the category with the smallest angle with the vehicle orientation as the vehicle orientation category.

[0200] When the detected vehicle is in different lanes, the angles between the structure lines and the absolute length and ratio of the structure lines will change accordingly. For vehicles in different directions in the same lane, the structure line information is also completely different. The structure line information can include information that can determine the direction of vehicles in different lanes.

[0201] The classification model in this embodiment can use a pruned dual path network (DPN) as a classifier to classify images. The DPN network has high parameter efficiency and can achieve better accuracy than other convolutional neural networks at the same scale.

[0202] In summary, in this embodiment, the vehicle's orientation information can actually be a continuous variable, but in most practical application scenarios, particularly high orientation accuracy is not required. Dividing the continuous orientation into multiple orientation categories can meet most application scenarios and simplify the vehicle orientation processing process, avoid unnecessary calculations, and improve processing efficiency.

[0203] Figure 7 A schematic diagram of the structure of a vehicle-mounted terminal provided in an embodiment of the present invention. Figure 1 The vehicle terminal includes: a processor 710 and a camera device 720. The processor 710 includes: an image acquisition module, a region determination module, a ratio determination module, a feature extraction module, a structure line determination module and a direction determination module (not shown in the figure).

[0204] An image acquisition module, used to acquire a road image containing a vehicle to be detected;

[0205] An area determination module is used to determine a vehicle area and a license plate area where a vehicle to be detected is located in a road image;

[0206] A ratio determination module, used to determine the width-to-height ratio of the vehicle to be detected based on the relative position relationship between the vehicle area and the license plate area;

[0207] A feature extraction module is used to extract a feature map of the vehicle area through a feature extraction layer in a vehicle structure line detection model; wherein the feature map includes multiple semantic groups; each feature vector in each semantic group is obtained by superimposing an attention factor; the attention factor is determined according to the similarity between the feature vector and the global semantic features of all feature vectors in the semantic group;

[0208] A structure line determination module is used to regress the feature map according to the aspect ratio through the regression layer in the vehicle structure line detection model to determine the structure line information of the vehicle to be detected in the vehicle area;

[0209] The orientation determination module is used to determine the orientation information corresponding to the structure line information according to the preset correspondence between the orientation information and the structure line information, as the orientation information of the vehicle to be detected.

[0210] In another embodiment of the present invention, based on Figure 7In the illustrated embodiment, the feature extraction layer includes: a convolution layer, an attention factor determination layer, and an attention factor superposition layer; the feature extraction module, when extracting a feature map of a vehicle area through the feature extraction layer in the vehicle structure line detection model, includes:

[0211] Through the convolution layer, an initial feature map of the vehicle area is extracted; wherein the initial feature map includes multiple semantic groups, and each semantic group includes multiple initial feature vectors;

[0212] Through the attention factor determination layer, for each semantic group, the global semantic feature is determined according to all the initial feature vectors in the semantic group, the initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature, and each initial attention factor is normalized to obtain the attention factor of each initial feature vector;

[0213] Through the attention factor superposition layer, the attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain the corresponding feature vector, and the feature map containing each feature vector is used as the feature map of the vehicle area.

[0214] In another embodiment of the present invention, based on Figure 7 In the illustrated embodiment, the feature extraction module determines the global semantic feature according to all the initial feature vectors in the semantic group, including:

[0215] According to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; where x i is the i-th initial feature vector in the semantic group χ, and m is the total number of all initial feature vectors in the semantic group χ;

[0216] The feature extraction module determines the initial attention factor of each initial feature vector according to the similarity between each initial feature vector and the global semantic feature, including:

[0217] According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where c i is the initial attention factor of the i-th initial feature vector;

[0218] The feature extraction module normalizes each initial attention factor to obtain the attention factor of each initial feature vector, including:

[0219] Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; is the attention factor of the i-th initial feature vector, ε is a constant;

[0220] The feature extraction module adds the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector, including:

[0221] According to the formula The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain the corresponding feature vector; where, σ is the sigmoid function gate, is the i-th feature vector, γ and β are the scaling coefficient and shift coefficient corresponding to the semantic group obtained by training the vehicle structure line detection model.

[0222] In another embodiment of the present invention, based on Figure 7 In the illustrated embodiment, the processor 710 further includes: a model training module (not shown in the figure), which is used to train the vehicle structure line detection model by using the following operations:

[0223] Acquire multiple sample vehicle images and a sample width-height ratio of a sample vehicle in each sample vehicle image, as well as standard structural line information corresponding to each sample vehicle image;

[0224] Inputting a sample vehicle image into a vehicle structure line detection model;

[0225] Determine a sample feature vector of the sample vehicle image through a first model parameter of a convolutional layer, and through an attention factor determination layer and an attention factor stacking layer;

[0226] Regressing the sample feature vector through the second model parameter of the regression layer and the sample width-height ratio corresponding to the sample vehicle image, to obtain reference structure line information of the sample vehicle in the sample vehicle image;

[0227] Calculate the difference between the reference structure line information and the corresponding standard structure line information;

[0228] When the difference is greater than a preset difference threshold, adjusting the first model parameter and the second model parameter according to the difference, and returning to perform the operation of inputting the sample vehicle image into the vehicle structure line detection model;

[0229] When the difference is less than a preset difference threshold, it is determined that the vehicle structure line detection model training is completed.

[0230] In another embodiment of the present invention, based on Figure 7 In the illustrated embodiment, the ratio determination module is specifically used for:

[0231] Determine virtual height information of the vehicle to be detected according to the number of pixels in the vehicle area in the height direction;

[0232] Determine the number of two pixels between the license plate region bounding box and the vehicle region bounding box in the width direction;

[0233] Determine virtual width information of the vehicle to be detected according to the smaller pixel number of the two pixel numbers;

[0234] The aspect ratio of the vehicle to be detected is determined according to the virtual height information and the virtual width information.

[0235] In another embodiment of the present invention, based on Figure 7 In the illustrated embodiment, the correspondence between the structure line information and the orientation information is: a correspondence between multiple orientation categories and structure line feature ranges; wherein each orientation category is a specific orientation selected from a set center point to a continuous orientation of a surrounding area;

[0236] The orientation determination module is specifically used for:

[0237] A structural line feature range corresponding to the structural line information is determined, and orientation information corresponding to the structural line information is determined according to an orientation classification corresponding to the determined structural line feature range.

[0238] The terminal embodiment and Figure 1 The method embodiments shown are embodiments based on the same inventive concept, and relevant parts can be referenced to each other. The above terminal embodiments correspond to the method embodiments, and have the same technical effects as the method embodiments. For specific descriptions, refer to the method embodiments.

[0239] Figure 8 A flowchart of a feature extraction method in a deep learning network provided in an embodiment of the present invention. The deep learning network can be any model trained using a deep learning method, such as a vehicle structural detection model. The deep learning network includes a convolutional neural network, etc. The method is applied to electronic devices. The method includes the following steps:

[0240] Step S810: Acquire an image whose features are to be extracted.

[0241] Step S820: extract features from the image according to preset feature extraction rules to obtain an initial feature map.

[0242] The initial feature map includes multiple semantic groups, and each semantic group includes multiple initial feature vectors.

[0243] Step S830: For each semantic group, determine the global semantic feature based on all the initial feature vectors in the semantic group, determine the initial attention factor of each initial feature vector based on the similarity between each initial feature vector and the global semantic feature, normalize each initial attention factor to obtain the attention factor of each initial feature vector.

[0244] Step S840: superimpose the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector, and use the feature map containing each feature vector as the feature map of the image.

[0245] In summary, in this embodiment, the deep learning network superimposes an attention factor in each feature vector when extracting image features. Using the attention factor to adjust the importance of the feature vector can enhance the detection features of the model, suppress the influence of similar patterns and noisy background in the image, and make the extracted image features better reflect the semantic features in the image.

[0246] Figure 8 The embodiment shown and Figure 1 The embodiments shown are based on the same inventive concept. Figure 8 The description of the illustrated embodiment can be found in Figure 1 The embodiment shown.

[0247] In another embodiment of the present invention, based on Figure 8 In the illustrated embodiment, step S830, i.e., for each semantic group, determining a global semantic feature according to all initial feature vectors in the semantic group, determining an initial attention factor of each initial feature vector according to the similarity between each initial feature vector and the global semantic feature, and normalizing each initial attention factor to obtain the attention factor of each initial feature vector, includes:

[0248] For each semantic group, according to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; wherein the x i is the i-th initial feature vector in the semantic group x, and m is the total number of all initial feature vectors in the semantic group x;

[0249] According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where the c i is the initial attention factor of the i-th initial feature vector;

[0250] Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; Said is the attention factor of the i-th initial feature vector, and ε is a constant.

[0251] In another embodiment of the present invention, based on Figure 8In the illustrated embodiment, step S840, the step of superimposing the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector includes:

[0252] According to the formula The attention factor of each initial feature vector is added to the corresponding initial feature vector to obtain the corresponding feature vector.

[0253] in, σ is the sigmoid function gate, is the i-th initial eigenvector x i The attention factor, is the i-th feature vector, γ and β are the scaling coefficient and shift coefficient corresponding to the semantic group obtained by training the deep learning network.

[0254] Fig. 9 A structural diagram of a feature extraction device in a deep learning network provided by an embodiment of the present invention. The device is applied to electronic equipment, and the device embodiment is similar to Figure 8 The method embodiment shown corresponds to the embodiment shown. The device includes:

[0255] An acquisition module 910 is configured to acquire an image to be extracted with features;

[0256] The extraction module 920 is configured to perform feature extraction on the image according to a preset feature extraction rule to obtain an initial feature map; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors;

[0257] The determination module 930 is configured to determine, for each semantic group, a global semantic feature according to all initial feature vectors in the semantic group, determine an initial attention factor of each initial feature vector according to a similarity between each initial feature vector and the global semantic feature; and normalize each initial attention factor to obtain an attention factor of each initial feature vector;

[0258] The superposition module 940 is configured to superimpose the attention factor of each initial feature vector onto the corresponding initial feature vector to obtain the corresponding feature vector; and use the feature map containing each feature vector as the feature map of the image.

[0259] In another embodiment of the present invention, based on Fig. 9 In the illustrated embodiment, the determination module 930 includes:

[0260] For each semantic group, according to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; wherein the x iis the i-th initial feature vector in the semantic group x, and m is the total number of all initial feature vectors in the semantic group x;

[0261] According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where the c i is the initial attention factor of the i-th initial feature vector;

[0262] Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; Said is the attention factor of the i-th initial feature vector, and ε is a constant.

[0263] In another embodiment of the present invention, based on Fig. 9 In the illustrated embodiment, the superposition module 940 includes:

[0264] According to the formula The attention factor of each initial feature vector is added to the corresponding initial feature vector to obtain the corresponding feature vector;

[0265] in, The σ is a sigmoid function gate, the is the i-th initial eigenvector x i The attention factor, is the i-th feature vector, and the γ and β are the scaling coefficient and shift coefficient corresponding to the semantic group obtained by training the deep learning network.

[0266] The above device embodiment corresponds to the method embodiment and has the same technical effect as the method embodiment. For specific description, please refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment part, which will not be repeated here.

[0267] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0268] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0269] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining vehicle orientation based on structure lines, characterized in that: include: Acquire a road image containing a vehicle to be detected; Determine a vehicle area and a license plate area where the vehicle to be detected is located in the road image; Determining the width-to-height ratio of the vehicle to be detected according to the relative position relationship between the vehicle area and the license plate area; A feature map of the vehicle area is extracted through a feature extraction layer in a vehicle structure line detection model; wherein the feature map includes a plurality of semantic groups; each feature vector in each semantic group is obtained by superimposing an attention factor; the attention factor is determined according to the similarity between the feature vector and the global semantic features of all feature vectors in the semantic group; the more similar the feature vector is to the global semantic features, the greater the attention factor of the feature vector; Regressing the feature map according to the aspect ratio through the regression layer in the vehicle structure line detection model to determine the structure line information of the vehicle to be detected in the vehicle area; According to the preset correspondence between the orientation information and the structure line information, determining the orientation information corresponding to the structure line information as the orientation information of the vehicle to be detected; The feature extraction layer includes: a convolution layer, an attention factor determination layer and an attention factor superposition layer; the step of extracting the feature map of the vehicle area through the feature extraction layer in the vehicle structure line detection model includes: Extracting an initial feature map of the vehicle area through the convolution layer; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors; By means of the attention factor determination layer, for each semantic group, a global semantic feature is determined according to all initial feature vectors in the semantic group, an initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature, and each initial attention factor is normalized to obtain an attention factor of each initial feature vector; The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector through the attention factor superposition layer to obtain the corresponding feature vector, and the feature map containing each feature vector is used as the feature map of the vehicle area.

2. The method according to claim 1, characterized in that The step of determining the global semantic feature according to all the initial feature vectors in the semantic group comprises: According to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; wherein the x i is the i-th initial feature vector in the semantic group x, and m is the total number of all initial feature vectors in the semantic group x; The step of determining the initial attention factor of each initial feature vector according to the similarity between each initial feature vector and the global semantic feature comprises: According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where the c i is the initial attention factor of the i-th initial feature vector; The step of normalizing each initial attention factor to obtain the attention factor of each initial feature vector includes: Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; where, Said is the attention factor of the i-th initial feature vector, and ε is a constant; The step of superimposing the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector includes: According to the formula The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain the corresponding feature vector; where, The σ is a sigmoid function gate, the is the i-th feature vector, and the γ and β are the scaling coefficient and the shift coefficient corresponding to the semantic group obtained by training the vehicle structure line detection model.

3. The method according to claim 1, characterized in that The vehicle structure line detection model is trained in the following manner: Acquire multiple sample vehicle images and a sample width-height ratio of a sample vehicle in each sample vehicle image, as well as standard structural line information corresponding to each sample vehicle image; Inputting the sample vehicle image into the vehicle structure line detection model; Determine a sample feature vector of the sample vehicle image through the first model parameter of the convolution layer, and through the attention factor determination layer and the attention factor stacking layer; Regressing the sample feature vector using the second model parameter of the regression layer and the sample width-height ratio corresponding to the sample vehicle image to obtain reference structure line information of the sample vehicle in the sample vehicle image; Calculating the difference between the reference structure line information and the corresponding standard structure line information; When the difference is greater than a preset difference threshold, adjusting the first model parameter and the second model parameter according to the difference, and returning to the step of inputting the sample vehicle image into the vehicle structure line detection model; When the difference is less than the preset difference threshold, it is determined that the vehicle structure line detection model training is completed.

4. The method according to claim 1, characterized in that The step of determining the aspect ratio of the vehicle to be detected based on the relative position relationship between the vehicle area and the license plate area comprises: Determining virtual height information of the vehicle to be detected according to the number of pixels in the height direction of the vehicle area; Determine the number of two pixels between the license plate region boundary box and the vehicle region boundary box in the width direction; Determining virtual width information of the vehicle to be detected according to the smaller number of pixels among the two numbers of pixels; The aspect ratio of the vehicle to be detected is determined according to the virtual height information and the virtual width information.

5. The method according to claim 1, characterized in that The correspondence between the structure line information and the orientation information is: a correspondence between a plurality of orientation categories and a structure line feature range; wherein each orientation category is a specific orientation selected from a set continuous orientation from a central point to a surrounding area; The step of determining the orientation information corresponding to the structure line information according to the preset correspondence between the orientation information and the structure line information comprises: A structural line feature range corresponding to the structural line information is determined, and orientation information corresponding to the structural line information is determined according to an orientation classification corresponding to the determined structural line feature range.

6. A feature extraction method in a deep learning network, characterized in that: include: Obtain an image to be feature extracted; According to a preset feature extraction rule, feature extraction is performed on the image to obtain an initial feature map; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors; For each semantic group, a global semantic feature is determined according to all initial feature vectors in the semantic group, and an initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature; each initial attention factor is normalized to obtain an attention factor of each initial feature vector; wherein the more similar the initial feature vector is to the global semantic feature, the greater the attention factor of the initial feature vector is; The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector to obtain the corresponding feature vector; and the feature map containing each feature vector is used as the feature map of the image.

7. The method according to claim 6, characterized in that For each semantic group, a global semantic feature is determined according to all initial feature vectors in the semantic group, and an initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature; The step of normalizing each initial attention factor to obtain the attention factor of each initial feature vector includes: For each semantic group, according to all the initial feature vectors in the semantic group and the formula Determine the global semantic feature g; wherein the x i is the i-th initial feature vector in the semantic group x, and m is the total number of all initial feature vectors in the semantic group x; According to the similarity between each initial feature vector and the global semantic feature, and formula c i =g·x i , determine the initial attention factor of each initial feature vector; where the c i is the initial attention factor of the i-th initial feature vector; Using formula Normalize each initial attention factor to obtain the attention factor of each initial feature vector; where, Said is the attention factor of the i-th initial feature vector, and ε is a constant.

8. The method according to claim 6 or 7, characterized in that The step of superimposing the attention factor of each initial feature vector to the corresponding initial feature vector to obtain the corresponding feature vector includes: According to the formula The attention factor of each initial feature vector is added to the corresponding initial feature vector to obtain the corresponding feature vector; in, The σ is a sigmoid function gate, the is the i-th initial eigenvector x i The attention factor, is the i-th feature vector, and the γ and β are the scaling coefficient and shift coefficient corresponding to the semantic group obtained by training the deep learning network.

9. A vehicle-mounted terminal, characterized in that: include: Processor and camera device; the processor includes: an image acquisition module, a region determination module, a ratio determination module, a feature extraction module, a structure line determination module and a direction determination module; The image acquisition module is used to acquire a road image containing a vehicle to be detected; The area determination module is used to determine the vehicle area and license plate area where the vehicle to be detected is located in the road image; The ratio determination module is used to determine the width-to-height ratio of the vehicle to be detected according to the relative position relationship between the vehicle area and the license plate area; The feature extraction module is used to extract a feature map of the vehicle area through a feature extraction layer in a vehicle structure line detection model; wherein the feature map includes multiple semantic groups; each feature vector in each semantic group is obtained by superimposing an attention factor; the attention factor is determined according to the similarity between the feature vector and the global semantic features of all feature vectors in the semantic group; the more similar the feature vector is to the global semantic features, the greater the attention factor of the feature vector; the structure line determination module is used to regress the feature map according to the width-to-height ratio through a regression layer in the vehicle structure line detection model to determine the structure line information of the vehicle to be detected in the vehicle area; The orientation determination module is used to determine the orientation information corresponding to the structure line information according to the corresponding relationship between the preset orientation information and the structure line information, as the orientation information of the vehicle to be detected; The feature extraction layer includes: a convolution layer, an attention factor determination layer and an attention factor superposition layer; the feature extraction module, when extracting the feature map of the vehicle area through the feature extraction layer in the vehicle structure line detection model, includes: Extracting an initial feature map of the vehicle area through the convolution layer; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors; By means of the attention factor determination layer, for each semantic group, a global semantic feature is determined according to all initial feature vectors in the semantic group, an initial attention factor of each initial feature vector is determined according to the similarity between each initial feature vector and the global semantic feature, and each initial attention factor is normalized to obtain an attention factor of each initial feature vector; The attention factor of each initial feature vector is superimposed on the corresponding initial feature vector through the attention factor superposition layer to obtain the corresponding feature vector, and the feature map containing each feature vector is used as the feature map of the vehicle area.

10. A feature extraction device in a deep learning network, characterized in that: include: An acquisition module is configured to acquire an image to be feature extracted; An extraction module is configured to perform feature extraction on the image according to a preset feature extraction rule to obtain an initial feature map; wherein the initial feature map includes a plurality of semantic groups, and each semantic group includes a plurality of initial feature vectors; A determination module is configured to determine, for each semantic group, a global semantic feature according to all initial feature vectors in the semantic group, and determine an initial attention factor of each initial feature vector according to the similarity between each initial feature vector and the global semantic feature; normalize each initial attention factor to obtain an attention factor of each initial feature vector; wherein the more similar the initial feature vector is to the global semantic feature, the greater the attention factor of the initial feature vector; The superposition module is configured to superimpose the attention factor of each initial feature vector onto the corresponding initial feature vector to obtain the corresponding feature vector; and use the feature map containing each feature vector as the feature map of the image.

Images