Object recognition method and apparatus, storage medium, and electronic device

Abstract

The application discloses an object recognition method and device, a storage medium and an electronic device. The method comprises the following steps: using a trained object recognition network to perform multiple times of recognition processing on a road image containing an object to be processed, and obtaining multiple pieces of object recognition information; wherein the object recognition information at least contains predicted key point information and category information obtained after the object to be processed is predicted; the object recognition network is obtained based on the difference information between the predicted key point information of a historical object in a sample road image and corresponding labeled key point information; and based on the multiple pieces of object recognition information, the object category, position information and key point obtained after the object to be processed is recognized are determined. The application solves the technical problem of low vehicle recognition efficiency caused by the multi-stage recognition technology in the related art.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically, to an object recognition method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] With economic development and improved infrastructure, the number of vehicles on the road is gradually increasing, placing a heavy burden on traffic supervision. In some emergency scenarios, such as highways, delayed processing of vehicle violations (crossing lanes) or stopped vehicles creates potential hazards for secondary accidents. For issues like crossing lanes, accurate location using only vehicle bounding boxes is difficult; therefore, utilizing vehicle key point localization is crucial. Currently, key point localization for targets in traffic scenarios mainly employs a two-stage method. First, detection algorithms detect road targets, obtaining corresponding bounding boxes and categories. Finally, based on the results of the bounding boxes, the vehicle's key points are located to obtain the final result. This multi-stage method not only increases the complexity of practical engineering deployment but also reduces overall performance. Furthermore, when the detected vehicle targets are incomplete or the bounding boxes contain other vehicle targets, vehicle key point localization errors can occur.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides an object recognition method and apparatus, a storage medium and an electronic device, to at least solve the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition techniques in related technologies.

[0005] According to one aspect of the present invention, an object recognition method is provided, comprising: performing multiple recognition processes on a road image containing an object to be processed using a trained object recognition network to obtain multiple object recognition information; wherein the object recognition information includes at least predicted key point information and category information obtained after predicting the object to be processed; the object recognition network is trained based on the difference information between predicted key point information and corresponding labeled key point information of historical objects in sample road images; and determining the object category, location information and key points obtained after recognizing the object to be processed based on the multiple object recognition information.

[0006] According to another aspect of the present invention, an object recognition device is also provided, comprising: a recognition unit, configured to perform multiple recognition processes on a road image containing an object to be processed using a trained object recognition network to obtain multiple object recognition information; wherein the object recognition information includes at least predicted key point information and category information obtained after predicting the object to be processed; and a determination unit, configured to: train the object recognition network based on the difference information between predicted key point information and corresponding labeled key point information of historical objects in sample road images; and determine the object category, location information, and key points obtained after recognizing the object to be processed based on the multiple object recognition information.

[0007] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the above-described object recognition method at runtime.

[0008] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the object recognition method described above through the computer program.

[0009] In this embodiment of the invention, a trained object recognition network is used to perform multiple recognition processes on a road image containing the object to be processed, resulting in multiple object recognition information. The object recognition information includes at least predicted keypoint information and category information obtained after predicting the object to be processed. The object recognition network is trained based on the difference information between the predicted keypoint information and the corresponding labeled keypoint information of historical objects in the sample road image. Based on multiple object recognition information, the method for determining the object category, location information, and keypoints obtained after recognizing the object to be processed is determined. This achieves the goal of accurately and timely recognizing vehicles and vehicle keypoints during road image detection, thereby improving the efficiency of vehicle, pedestrian, and vehicle keypoint detection. This solves the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition techniques in related technologies. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0011] Figure 1 This is a schematic diagram of an application environment for an optional object recognition method according to an embodiment of the present invention;

[0012] Figure 2This is a schematic diagram of an application environment for another optional object recognition method according to an embodiment of the present invention;

[0013] Figure 3 This is a flowchart illustrating an optional object recognition method according to an embodiment of the present invention;

[0014] Figure 4 This is a schematic diagram of the interface display of an optional object recognition method according to an embodiment of the present invention;

[0015] Figure 5 This is a flowchart illustrating an optional object recognition method according to an embodiment of the present invention.

[0016] Figure 6 This is a flowchart illustrating another optional object recognition method according to an embodiment of the present invention;

[0017] Figure 7 This is a flowchart illustrating another optional object recognition method according to an embodiment of the present invention;

[0018] Figure 8 This is a flowchart illustrating another optional object recognition method according to an embodiment of the present invention;

[0019] Figure 9 This is a schematic diagram of the structure of an optional object recognition device according to an embodiment of the present invention;

[0020] Figure 10 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] First, the key terms involved in the embodiments of this application will be explained: NMS algorithm: Non-maximum suppression algorithm. NMS is widely used in object detection algorithms. Its purpose is to eliminate redundant candidate boxes and find the best object detection position.

[0024] According to one aspect of the present invention, an object recognition method is provided. Optionally, as an alternative implementation, the object recognition method may be applied to, but is not limited to, [examples of other methods]. Figure 1 In the environment shown.

[0025] Figure 1 In this system, terminal device 104 is responsible for human-computer interaction with user 102. Terminal device 104 includes a memory 106, a processor 108, and a display 110. Terminal device 104 can interact with management platform 114 via network 112. Server 114 includes a database 116 and a processing engine 118. First image acquisition device 120 interacts with terminal device 104 via network 112. Image acquisition device 120 can be connected to network 112 via a wired or wireless module. Terminal device 104 can display the video or image acquired by first image acquisition device 120 via display 110. After acquiring the images or videos acquired by first image acquisition device 120 and second image acquisition device 122, management platform 114 stores the images or videos in database 116. Processing engine can compare the images or videos acquired by first image acquisition device 120.

[0026] As another optional implementation, the object recognition method described above in this application can be applied to... Figure 2 In the middle. For example Figure 2 As shown, user 202 and user equipment 204 can interact. User equipment 204 includes a memory 206 and a processor 208. In this embodiment, user equipment 204 can, but is not limited to, referencing and executing the operations performed by the terminal device 104 to achieve road map recognition and early warning.

[0027] Optionally, in this embodiment, the terminal device 104 and user device 204 can be terminal devices configured with a target client, including but not limited to at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), laptops, tablets, PDAs, MIDs (Mobile Internet Devices), PADs, desktop computers, smart TVs, etc. The target client can be a video client, instant messaging client, browser client, educational client, etc. The first image acquisition device 120 can be a camera configured with a wired or wireless module. The network 112 can include, but is not limited to, wired networks and wireless networks. The wired network includes local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). The wireless network includes Bluetooth, Wi-Fi, and other networks that enable wireless communication. The server 114 can be a single server, a server cluster consisting of multiple servers, or a cloud server. The above is merely an example, and no limitations are imposed in this embodiment.

[0028] Alternatively, as an alternative implementation method, such as Figure 3 As shown, the above object recognition method includes:

[0029] S302, using the trained object recognition network, the road image containing the object to be processed is processed multiple times to obtain multiple object recognition information; wherein, the object recognition information includes at least the predicted key point information and category information obtained after predicting the object to be processed; the object recognition network is trained based on the difference information between the predicted key point information and the corresponding labeled key point information of historical objects in the sample road image.

[0030] S304, based on multiple object recognition information, determine the object category, location information and key points obtained after recognizing the object to be processed.

[0031] Optionally, in this embodiment, the above object recognition method can be applied, but is not limited to, to the inspection process of pedestrians, motor vehicles, and non-motor vehicles on a road map.

[0032] In this embodiment, as Figure 4 As shown, the image acquisition device captures each video frame of the road image and identifies each object in the image to obtain the output image. In the output image, pedestrians are marked and displayed using pedestrian anchor frame 402, non-motorized vehicles are marked and displayed using anchor frame 404, motorized vehicles are marked and displayed using anchor frame 406, vehicle headlights are marked and displayed using anchor frame 406a, vehicle rearview mirrors are marked and displayed using anchor frame 406b, and vehicle license plates are marked and displayed using anchor frame 406c.

[0033] It is understood that in this embodiment, after the above-mentioned object to be processed is input into the trained model, multiple output values ​​(out) are obtained, and then the final category, detection box and key points of the target are determined based on the multiple output values ​​(out).

[0034] As an optional approach, when the location information includes the location of the detection box of the object to be processed, the object category, location information, and key points obtained after recognizing the object to be processed are determined based on multiple object recognition information, including:

[0035] S1, using the non-maximum suppression algorithm, the predicted key point information and category information in each object recognition information are processed to obtain the object category of the object to be processed, the location of the detection box of the object to be processed, and the target key point information of the object to be processed. The target key point information is determined based on the predicted key point information contained in multiple object recognition information.

[0036] S2, based on the key point offset information, processes the target key point information to obtain the key points of the object to be processed.

[0037] Optionally, the aforementioned target key point information includes key point coordinates; key point offset information includes coordinate offset relative to the key point coordinates; wherein, based on the key point offset information, the target key point information is processed to obtain the key points of the object to be processed, including: superimposing the key point coordinates and the coordinate offset, and determining the position point corresponding to the superimposed coordinates as the key point of the object to be processed.

[0038] It should be noted that in this embodiment, the NMS algorithm is used to process the above-mentioned out information to obtain the final recognition result. The target key point information is the coordinate information obtained by processing using the following formula:

[0039]

[0040] Among them, c x and c y Corresponding to the above keypoint offset information, t x and t y Corresponding to the above target key point information. (land) x and land y It is the true x-coordinate of the key point; during the prediction process, there are multiple key points in the image, corresponding to multiple lands. x and land y ;c x and c y It is a unique value, where c is the only value. x It is a grid; the coordinate t of the top left corner of each grid.x It is a normalized value between 0 and 1, used to determine the specific location of the network.

[0041] Through the above embodiments of this application, by superimposing the key point coordinates and the coordinate offset, the position point corresponding to the superimposed coordinates is determined as the key point of the object to be processed. This achieves the goal of accurately and timely identifying vehicles and vehicle key points during road image detection, thereby improving the technical effect of vehicle, pedestrian and vehicle key point detection efficiency. It also solves the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition technology in related technologies.

[0042] As an optional approach, the object recognition network described above includes N sub-recognition networks. Each sub-recognition network contains a feature extraction layer and a recognition layer connected in series, and the feature extraction layer of the i-th sub-recognition network is connected to the recognition layer of the (i+1)-th sub-recognition network. N is an integer greater than 1, and i is an integer greater than 0 and less than N.

[0043] Optionally, the trained object recognition network is used to perform multiple recognition processes on the road image containing the object to be processed to obtain multiple object recognition information, including: using N sub-recognition networks to recognize the road image to obtain N object recognition information, with each sub-recognition network corresponding to one object recognition information.

[0044] like Figure 7 As shown, the feature extraction layer included in the above sub-recognition network can be... Figure 7 The Block 1 form shown in Figure (a) appears in the above sub-recognition network, and the above recognition layer can be... Figure 7 The DetectHead form shown in Figure (b) appears in the above sub-recognition network.

[0045] Through the above embodiments, the trained object recognition network is used to perform multiple recognition processes on a road image containing the object to be processed, thereby obtaining multiple object recognition information. This includes: using the N sub-recognition networks to recognize the road image to obtain N object recognition information, with each sub-recognition network corresponding to one object recognition information, thereby achieving accurate recognition of the object to be recognized and improving the accuracy of object recognition.

[0046] As an optional approach, the above-mentioned methods utilize N sub-recognition networks to identify road images, obtaining N object recognition information, including:

[0047] S1, the process of recognizing the road image using the j-th sub-recognition network includes:

[0048] S2, using the feature extraction layer in the j-th sub-recognition network, the features output by the feature extraction layer of the (j-1)-th sub-recognition network are processed to obtain the features output by the feature extraction layer in the j-th sub-recognition network;

[0049] S3, using the recognition layer in the j-th sub-recognition network, the features output by the feature extraction layer in the j-th sub-recognition network are recognized to obtain the object recognition information corresponding to the j-th sub-recognition network; where: j is an integer greater than 0 and not greater than N; when j is 1, the features output by the feature extraction layer of the (j-1)-th sub-recognition network are the preprocessed features of the road image.

[0050] like Figure 8 As shown in the figure, in this embodiment, there are four Block 1 blocks used for concatenating image processing, and three corresponding Detect Heads are shown. The Detect Heads corresponding to the second to fourth Block 1 blocks are Detect Head_1, Detect Head_2, and Detect Head_3, respectively. It should be noted that the preprocessed features obtained through the above processing method are the features obtained from preliminary feature extraction.

[0051] The process of recognizing the road image using the j-th sub-recognition network, as described in the above embodiments of this application, includes: processing the features output by the feature extraction layer of the (j-1)-th sub-recognition network using the feature extraction layer in the j-th sub-recognition network to obtain the features output by the feature extraction layer of the j-th sub-recognition network; and recognizing the features output by the feature extraction layer of the j-th sub-recognition network using the recognition layer in the j-th sub-recognition network to obtain the object recognition information corresponding to the j-th sub-recognition network. Wherein, j is an integer greater than 0 and not greater than N; when j is 1, the features output by the feature extraction layer of the (j-1)-th sub-recognition network are preprocessed features of the road image, thus achieving the technical effect of improving the efficiency of object recognition.

[0052] As an alternative, the object recognition network described above also includes a downsampling layer and a feature preprocessing layer, the feature preprocessing layer having the same network structure as the feature extraction layer contained in the sub-recognition network;

[0053] Optionally, before using N sub-recognition networks to recognize the road image and obtain N object recognition information, with each sub-recognition network corresponding to one object recognition information, the method further includes: using a downsampling layer to downsample the road image; and using a feature preprocessing layer to extract features from the structure obtained by the downsampling process to obtain preprocessed features of the road image.

[0054] It should be noted that, as Figure 8As shown in the figure, the first Block 1 corresponds to the aforementioned feature preprocessing layer. The aforementioned downsampling layer corresponds to the double-downsampling structure in the entire model. Specifically, the initial two layers in the network are a convolutional layer with a stride of 2 and a kernel size of 7, and a max pooling layer with a kernel size of 3 and a stride of 2, which are downsampled by 2. The number of output channels of the convolutional layer is 64.

[0055] Through the above embodiments of this application, by using the downsampling layer to downsample the road image, and using the feature preprocessing layer to extract features from the structure obtained by the downsampling process, the preprocessed features of the road image are obtained, thereby improving the efficiency of vehicle and pedestrian and vehicle key point detection, and thus solving the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition technology in related technologies.

[0056] As an alternative approach, the feature extraction layer includes a dense layer and a transition layer connected in series. The dense layer includes K dense blocks, each containing a convolutional layer with a first kernel size and a convolutional layer with a second kernel size. The transition layer includes a first max pooling layer with a third kernel size.

[0057] Optionally, the aforementioned recognition layer includes a detection head structure and a detection output structure. The detection head structure includes a second max pooling layer, a detection convolutional layer, and a global processing layer. The first and second detection convolutional layers in the detection convolutional layer have the same kernel size but different strides and output channel numbers. The global processing layer includes a global max pooling layer, a first processing convolutional layer, and a second processing convolutional layer. The first and second processing convolutional layers have the same kernel size and stride but different output channel numbers. The output of the detection convolutional layer and the output of the global processing layer are multiplied to obtain a first result. The second max pooling layer and the first result are added to obtain a second result. The detection output structure includes a first output convolutional layer and a second output convolutional layer. The first and second output convolutional layers have the same kernel size and stride but different output channel numbers. The second result is input into the detection output structure.

[0058] Through the above embodiments of this application, the output result of the detection convolutional layer and the output result of the global processing layer are multiplied to obtain a first operation result; the second max pooling layer and the first operation result are added to obtain a second operation result; the detection output structure includes a first output convolutional layer and a second output convolutional layer, wherein the first output convolutional layer and the second output convolutional layer have the same convolutional kernel size and stride, but different numbers of output channels; the second operation result is input into the detection output structure, thereby achieving the technical effect of improving the detection efficiency of vehicles, pedestrians and vehicle key points, and thus solving the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition technology in related technologies.

[0059] As an optional approach, the object recognition information obtained in each of the above rounds carries the current output multidimensional vector trained in the current round. The current output multidimensional vector is:

[0060] out = a(x1 + c + x2 + n(y1 + y2))

[0061] Where out is the current output multidimensional vector, a is the number of detection boxes for the object to be processed, x1 is the confidence score, c is the target category indicator coefficient, x2 is the coordinate information of the key point, n is the number of key points, y1 is the number of coordinate axes, and y2 is the number of vehicles used to indicate the target.

[0062] As an optional approach, the aforementioned object recognition network is trained as follows: the object recognition network is iteratively trained multiple times until the training termination condition is met, resulting in the trained object recognition network; wherein, one round of iterative training includes:

[0063] S1, using the object recognition network in training, performs multiple recognition processes on the road image samples to obtain multiple object recognition information of the sample road images;

[0064] S2, based on the difference between the predicted key point information and the corresponding labeled key point information of historical objects in multiple object recognition information of sample road images, adjust the model parameters in the object recognition network during training.

[0065] Through the above-described embodiments of this application, the object recognition network under training is iteratively trained multiple times until the training termination condition is met, resulting in a trained object recognition network. Each iterative training round includes: using the trained object recognition network to perform multiple recognition processes on road image samples, obtaining multiple object recognition information for the sample road images; based on the differences between the predicted keypoint information and the corresponding labeled keypoint information of historical objects in the multiple object recognition information of the sample road images, the model parameters in the trained object recognition network are adjusted, thereby achieving the training of a network for object recognition. This improves the efficiency of vehicle and pedestrian keypoint detection, thus solving the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition techniques in related technologies.

[0066] As an optional approach, the difference between the predicted keypoint information and the corresponding labeled keypoint information of historical objects in the above-mentioned object recognition information based on multiple object recognition information of sample road images is used to adjust the model parameters in the trained object recognition network, including:

[0067] S1, determine the first prediction difference value corresponding to each historical object; wherein, the prediction difference value is the difference between the prediction key point information and the corresponding labeled key point information of the corresponding historical object.

[0068] S2, Based on the determined first prediction difference values, determine the comprehensive prediction difference value;

[0069] S3 uses the comprehensive prediction difference value to adjust the model parameters in the object recognition network during training.

[0070] As an optional approach, before determining the first predicted difference value for each historical object, the following steps are also included:

[0071] S1, in response to the historical object's size being smaller than a first preset size threshold, amplify the first predicted difference value of the historical object; and / or

[0072] S2, in response to the historical object's size being greater than a second preset size threshold, the first predicted difference value of the historical object is reduced.

[0073] As an optional approach, the above-mentioned amplification of the first predicted difference value of historical objects includes:

[0074] The first predicted difference value is amplified using a loss function constructed based on the ratio of the target's relative feature map; or

[0075] Based on the first reference value, the first predicted difference values ​​of historical objects are weighted and processed, where the first reference value is a value greater than 1; or

[0076] Divide the first predicted difference value of the historical object by the second reference value, which is a positive number less than 1.

[0077] As an optional approach, the above-mentioned reduction of the first predicted difference value of historical objects includes:

[0078] Subtract the target constant from the first predicted difference value to perform a reduction process; or

[0079] Based on the third reference value, the first prediction difference value of historical objects is weighted, where the third reference value is less than 1; or

[0080] Divide the first predicted difference value of the historical object by the fourth reference value, which is a positive number greater than 1.

[0081] As an optional approach, the category identifier and location identifier of the target object in the output target object image can be specifically obtained using the following methods:

[0082] S1: Collect traffic scene materials and label them according to pedestrian, non-motorized vehicle and motorized vehicle categories. The label includes the border of the target and the corresponding category.

[0083] S2, based on the labeled borders of the collected materials, calculate K cluster centers using the k-means++ clustering algorithm. Each center is composed of height and width.

[0084] S3, the image is processed through the designed lightweight network to obtain three feature outputs, each feature output containing out channels; note: the width and height values ​​of the output are scaled relative to the anchor box;

[0085] S4 multiplies the width and height scaling values ​​obtained in S3 with the anchor frame size, then performs a non-maximum operation, and filters the results based on the set IOU threshold and category execution threshold to obtain the final output category identifier and location identifier.

[0086] In this embodiment of the invention, a trained object recognition network is used to perform multiple recognition processes on a road image containing the object to be processed, resulting in multiple object recognition information. The object recognition information includes at least predicted keypoint information and category information obtained after predicting the object to be processed. The object recognition network is trained based on the difference information between the predicted keypoint information and the corresponding labeled keypoint information of historical objects in the sample road image. Based on multiple object recognition information, the method for determining the object category, location information, and keypoints obtained after recognizing the object to be processed is determined. This achieves the goal of accurately and timely recognizing vehicles and vehicle keypoints during road image detection, thereby improving the efficiency of vehicle, pedestrian, and vehicle keypoint detection. This solves the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition techniques in related technologies.

[0087] Based on the above embodiments, in one application embodiment, the above object recognition method includes: Step S502, annotating and preprocessing the bounding boxes and key points of the acquired image to obtain an image containing multiple targets; Step S504, inputting the image into a lightweight feature extractor to obtain a first feature map, a second feature map, and a third feature map; Step S506, assigning each target in the image to the extracted first feature map, second feature map, and third feature map according to the size of the image annotation box and anchor box; Step S508, regressing the position and category information of the corresponding key points of the target based on the feature map including the target feature points, and updating the gradient in reverse; Step S510, using the Non-Maximum Suppression (NMS) algorithm to filter out redundant boxes, obtaining the target regression points on each feature map, and calculating the target bounding box and category based on the point, i.e., the key point information and category of the target.

[0088] This proposal is based on the YOLO object detection algorithm structure and provides an end-to-end design for multi-object bounding box, key point detection and recognition methods.

[0089] S602 involves labeling the collected traffic scene materials, categorizing them as pedestrians, motor vehicles, and non-motor vehicles, and labeling n key points of motor vehicle targets (e.g., buses, trucks, cars, etc.). The proposal uses 8 key points: left front light, right front light, front license plate, left rear light, right rear light, rear license plate, left rear light, and right rear light. The labeled data is clustered using the k-means++ clustering algorithm to obtain a anchor boxes. In this proposal, the number of anchor boxes is set to 3, and the labeled target positions and width and height information are normalized.

[0090] S604, such as Figure 5 , Figure 6 , Figure 7 and Figure 8As shown, the processed image is input into the designed lightweight neural network. Given the excellent performance of the DenseNet network, this scheme uses it as the basic structure and combines it with the characteristics of the CSP (Cross Stage Partial) structure. Since there are many small targets in traffic scenes, the network input size is 480 x 800. The lightweight network structure diagram of this proposal is shown below. Figure 2 As shown, the specific parameters are as follows: First, the initial two layers of the network are designed as a convolutional layer with a stride of 2 and a kernel size of 7, and a max pooling layer with a kernel size of 3 and a stride of 2, with a 2x downsampling. The number of output channels of the convolutional layer is 64. Then, three Dense Blocks are cascaded. Each Dense Block consists of a convolutional layer with a kernel size of 1 and 48 output channels, and a convolutional layer with a kernel size of 3 and 24 output channels. A Transition Block is also stored afterward, consisting of a max pooling layer with a kernel size of 1 and a kernel size of 2 and a stride of 2. The compression ratio used in the design is 0.45, so the number of output channels of the Transition Layer is 0.45 times that of the previous layer. Then, max pooling layers with kernel sizes of 13x13, 9x9, and 5x5, and a global max pooling layer are connected at the channel level to obtain the Block1 module. Afterward, the network cascades three more groups of Block1 modules, each corresponding to a Dense Block. The number of blocks is 4, 6 and 4 respectively. The first group of Dense Blocks has 64 output channels with a kernel size of 1 and 32 output channels with a kernel size of 3.

[0091] The second group of Dense Blocks has 96 output channels with a kernel size of 1 and 48 output channels with a kernel size of 3. The third group of Dense Blocks has 128 output channels with a kernel size of 1 and 64 output channels with a kernel size of 3. The Transition Block module is removed after the last Dense Block. The outputs of the first, second, and third Transition Blocks are used as the inputs to the respective Detect heads. The head structure for object detection is as follows: the obtained feature maps are passed through a max pooling layer with a stride of 2, a convolutional layer with a kernel size of 3, 128 channels, and a stride of 2, and a global max pooling layer to obtain three outputs. The output of the convolutional layer with a stride of 2 is then passed through a convolutional layer with a kernel size of 1, a stride of 1, and 64 channels. The output of the max pooling layer is then passed through two convolutional layers with a stride of 1, a kernel size of 3, and 128 and 64 channels respectively. Finally, a sigmoid layer is passed through the sigmoid layer, and this part of the output is multiplied by the output of the convolutional layer with a stride of 2. The algorithm performs computation and connects to a max pooling layer with a stride of 2 at the channel level. Finally, the final output is obtained through a convolutional layer with a kernel size of 1, a stride of 3, and 128 channels, and a classification and regression convolutional layer with a kernel size of 1, a stride of 3, and out channels, where out = a × (1 + c + 4 + n × (2 + 1)), where a is the number of anchor boxes and c is the target category. Given the uncertain nature of the target keypoint regression direction, a linear function is used instead of a linear function to improve performance compared to the target center point. The width of the network is designed by adding max pooling layers with kernel sizes of 13x13, 9x9, and 5x5 and a global pooling layer after each Transition Block module. A CSP structure is introduced into the detection head, and a Detect Head module is designed in conjunction with the channel attention method to extract useful information from different features in the high-level layers. At the same time, this proposal designs the network parameters in detail to achieve real-time inference on mobile devices.

[0092] S606, the outputs of the first, second, and third groups of category regression layers in step 2 are filtered using the NMS algorithm and the preset threshold values ​​in the scheme to obtain the regression target center points of each group of category regression layers. Based on this, the target category, border center point, border width and height, regression key point positions, and their category information are calculated. The formula for calculating the target key point positions is as follows:

[0093] land x =c x +t x

[0094] land y =c y +ty

[0095] When calculating the target keypoint loss, i.e., during sample mining, the offset loss of each keypoint is calculated for each feature point assigned to the target. Target assignment mainly involves first assigning a regression point to each positive sample in the image on the feature map. For smaller targets, multiple regression points are defined on different feature maps. When the IoU value between the target bounding box and the anchor box is less than a set threshold, that point on the feature map is defined as a negative sample. Since the target keypoints are small and have limited semantic information, this scheme improves the existing position regression loss function to obtain an effective keypoint position regression loss function, as shown in the following formula:

[0096]

[0097] Where loss is the output value of the loss function, ω is the divergence amount greater than 1, w and h represent the proportion of the target relative to the feature map, β is a constant in [1,∞), and x is the position indicated by the above position marker.

[0098] Where w and h represent the relative proportions of the target to the feature map, which are constants. When the detected target is small, its position error is amplified to a certain extent to improve the positioning accuracy of key points of small targets.

[0099] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0100] According to another aspect of the present invention, an object recognition apparatus for implementing the above-described object recognition method is also provided. For example... Figure 9 As shown, the device includes:

[0101] The recognition unit 902 is used to perform multiple recognition processes on a road image containing an object to be processed using a trained object recognition network to obtain multiple object recognition information; wherein, the object recognition information includes at least predicted key point information and category information obtained after predicting the object to be processed; the object recognition network is trained based on the difference information between the predicted key point information and the corresponding labeled key point information of historical objects in the sample road image.

[0102] The determining unit 904 is used to determine the object category, location information and key points obtained after recognizing the object to be processed based on multiple object recognition information.

[0103] In this embodiment of the invention, a trained object recognition network is used to perform multiple recognition processes on a road image containing the object to be processed, resulting in multiple object recognition information. The object recognition information includes at least predicted keypoint information and category information obtained after predicting the object to be processed. The object recognition network is trained based on the difference information between the predicted keypoint information and the corresponding labeled keypoint information of historical objects in the sample road image. Based on multiple object recognition information, the method for determining the object category, location information, and keypoints obtained after recognizing the object to be processed is determined. This achieves the goal of accurately and timely recognizing vehicles and vehicle keypoints during road image detection, thereby improving the efficiency of vehicle, pedestrian, and vehicle keypoint detection. This solves the technical problem of low vehicle recognition efficiency caused by the use of multi-stage recognition techniques in related technologies.

[0104] According to another aspect of the present invention, an electronic device for implementing the above-described object recognition method is also provided. This electronic device may be... Figure 1 The terminal device or server shown. For example... Figure 10 As shown, the electronic device includes a memory 1002 and a processor 1004. The memory 1002 stores a computer program, and the processor 1004 is configured to execute the steps of any of the above method embodiments via the computer program.

[0105] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0106] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0107] S1, using the trained object recognition network, multiple recognition processes are performed on the road image containing the object to be processed to obtain multiple object recognition information; among them, the object recognition information includes at least the predicted key point information and category information obtained after predicting the object to be processed; the object recognition network is trained based on the difference information between the predicted key point information and the corresponding labeled key point information of historical objects in the sample road image.

[0108] S2, based on multiple object recognition information, determines the object category, location information and key points obtained after recognizing the object to be processed.

[0109] Alternatively, as those skilled in the art will understand, Figure 10The structure shown is for illustrative purposes only. Electronic devices or electronic equipment can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other electronic devices. Figure 10 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 10 The different configurations shown.

[0110] The memory 1002 can be used to store software programs and modules, such as the program instructions / modules corresponding to the object recognition method and device in this embodiment of the invention. The processor 1004 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002, thereby realizing the object recognition method described above. The memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1002 may further include memory remotely located relative to the processor 1004, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1002 may be used, but is not limited to, to store information such as video or images of virtual security inspection objects. As an example, such as Figure 10 As shown, the memory 1002 may include, but is not limited to, the identification unit 902 and the determination unit 904 in the object identification device. Furthermore, it may include, but is not limited to, other module units in the object identification device, which will not be elaborated upon in this example.

[0111] Optionally, the transmission device 1010 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1010 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1010 is a radio frequency (RF) module, used for wireless communication with the Internet.

[0112] In addition, the aforementioned electronic device also includes: a display 1008 for displaying video or image information of the security inspection object; and a connection bus 1010 for connecting various module components in the aforementioned electronic device.

[0113] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0114] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0115] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0116] S1, using the trained object recognition network, multiple recognition processes are performed on the road image containing the object to be processed to obtain multiple object recognition information; among them, the object recognition information includes at least the predicted key point information and category information obtained after predicting the object to be processed; the object recognition network is trained based on the difference information between the predicted key point information and the corresponding labeled key point information of historical objects in the sample road image.

[0117] S2, based on multiple object recognition information, determines the object category, location information and key points obtained after recognizing the object to be processed.

[0118] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0119] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0121] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An object recognition method, characterized in that, include: A trained object recognition network is used to perform multiple recognition processes on road images containing objects to be processed, resulting in multiple object recognition information. The object recognition information includes at least predicted keypoint information and category information obtained after predicting the objects to be processed. The object recognition network is trained based on the difference between predicted keypoint information and corresponding labeled keypoint information of historical objects in sample road images. The object recognition network includes N sub-recognition networks. Each sub-recognition network contains a recognition layer. The recognition layer includes a detection head structure and a detection output structure. The detection head structure includes a second max-pooling layer, a detection convolutional layer, and a global detection layer. The processing layer includes a first detection convolutional layer and a second detection convolutional layer with the same kernel size but different strides and output channels. The global processing layer includes a global max-pooling layer, a first processing convolutional layer, and a second processing convolutional layer. The first and second processing convolutional layers have the same kernel size and stride but different output channels. The output of the detection convolutional layer is multiplied by the output of the global processing layer to obtain a first result. The second max-pooling layer is added to the first result to obtain a second result. The second result is then input into the detection output structure. Based on the multiple object recognition information, the object category, location information, and key points obtained after recognizing the object to be processed are determined.

2. The method according to claim 1, characterized in that, When the location information includes the location of the detection box of the object to be processed, the step of determining the object category, location information, and key points obtained after recognizing the object to be processed based on the multiple object recognition information includes: Using a nonmaximum suppression algorithm, the predicted key point information and category information in each of the object recognition information are processed to obtain the object category of the object to be processed, the location of the detection box of the object to be processed, and the target key point information of the object to be processed. The target key point information is determined based on the predicted key point information contained in the multiple object recognition information. Based on the key point offset information, the target key point information is processed to obtain the key points of the object to be processed.

3. The method according to claim 2, characterized in that, The target key point information includes key point coordinates; the key point offset information includes coordinate offset relative to the key point coordinates; wherein, processing the target key point information based on the key point offset information to obtain the key points of the object to be processed includes: The key point coordinates and the coordinate offset are superimposed, and the position point corresponding to the superimposed coordinates is determined as the key point of the object to be processed.

4. The method according to claim 1, characterized in that, Each of the sub-identification networks contains a series-connected feature extraction layer, and the feature extraction layer of the i-th sub-identification network is connected to the feature extraction layer of the (i+1)-th sub-identification network, where N is an integer greater than 1 and i is an integer greater than 0 and less than N; Using a trained object recognition network, road images containing objects to be processed are subjected to multiple recognition processes to obtain multiple object recognition information, including: The road image is identified using the N sub-recognition networks respectively, resulting in N object recognition information, with each sub-recognition network corresponding to one object recognition information.

5. The method according to claim 4, characterized in that, The process of recognizing the road image using the N sub-recognition networks to obtain N object recognition information includes: The process of recognizing the road image using the j-th sub-recognition network includes: Using the feature extraction layer in the j-th sub-recognition network, the features output by the feature extraction layer of the (j-1)-th sub-recognition network are processed to obtain the features output by the feature extraction layer in the j-th sub-recognition network. Using the recognition layer in the j-th sub-recognition network, the features output by the feature extraction layer in the j-th sub-recognition network are recognized to obtain the object recognition information corresponding to the j-th sub-recognition network; wherein: The j is an integer greater than 0 and not greater than N; when j is 1, the feature output by the feature extraction layer of the (j-1)th sub-recognition network is the preprocessed feature of the road image.

6. The method according to claim 5, characterized in that, The object recognition network further includes a downsampling layer and a feature preprocessing layer, wherein the feature preprocessing layer has the same network structure as the feature extraction layer contained in the sub-recognition network. Before the step of recognizing the road image using the N sub-recognition networks to obtain N object recognition information, where each sub-recognition network corresponds to one object recognition information, the method further includes: The road image is downsampled using the downsampling layer. The feature preprocessing layer is used to extract features from the structure obtained by the downsampling process to obtain the preprocessed features of the road image.

7. The method according to claim 4, characterized in that, The feature extraction layer includes a dense layer and a transition layer connected in series. The dense layer includes K dense blocks, and each dense block includes a convolutional layer with a first kernel size and a convolutional layer with a second kernel size. The transition layer includes a first max pooling layer with a third kernel size.

8. The method according to claim 4, characterized in that, The detection output structure includes a first output convolutional layer and a second output convolutional layer, wherein the first output convolutional layer and the second output convolutional layer have the same kernel size and stride, but different numbers of output channels.

9. The method according to any one of claims 1 to 8, characterized in that, Each round of object recognition information carries the current output multidimensional vector trained in the current round, and the current output multidimensional vector is: out = a (x1 + c + x2 + n (y1 + y2)) Wherein, out is the current output multidimensional vector, a is the number of detection boxes of the object to be processed, x1 is the confidence score, c is the target category indicator coefficient, x2 is the coordinate information of the key point, n is the number of key points, y1 is the number of coordinate axes, and y2 is the number of vehicles used to indicate the target.

10. The method according to any one of claims 1 to 8, characterized in that, The object recognition network was trained in the following manner: The object recognition network under training is iterated multiple times until the training termination condition is met, resulting in the trained object recognition network; one round of iterative training includes: By using the object recognition network in training, the road image samples are processed multiple times to obtain multiple object recognition information of the sample road images; Based on the differences between the predicted keypoint information and the corresponding labeled keypoint information of historical objects in the multiple object recognition information of the sample road image, the model parameters in the trained object recognition network are adjusted.

11. The method according to claim 10, characterized in that, The method of adjusting the model parameters in the trained object recognition network based on the difference information between predicted keypoint information and corresponding labeled keypoint information of historical objects in the multiple object recognition information of the sample road image includes: Determine the first predicted difference value for each historical object; wherein, the predicted difference value is the difference between the predicted key point information and the corresponding labeled key point information of the corresponding historical object; Based on the determined first prediction difference values, the comprehensive prediction difference value is determined; The model parameters in the trained object recognition network are adjusted using the comprehensive prediction difference value.

12. The method according to claim 11, characterized in that, Before determining the first predicted difference value corresponding to each historical object, the method further includes: In response to the historical object's size being smaller than a first preset size threshold, the first predicted difference value of the historical object is amplified; and / or In response to the historical object's size being greater than a second preset size threshold, the first predicted difference value of the historical object is reduced.

13. The method according to claim 12, characterized in that, The amplification process for the first predicted difference value of the historical object includes: The first predicted difference value is amplified using a loss function constructed based on the ratio of the target's relative feature map; or Based on a first reference value, the first predicted difference value of the historical object is weighted and processed, where the first reference value is a value greater than 1; or Divide the first predicted difference value of the historical object by the second reference value, where the second reference value is a positive number less than 1.

14. The method according to claim 12, characterized in that, The process of reducing the first predicted difference value of the historical object includes: Subtract the target constant from the first predicted difference value to perform a reduction process; or Based on a third reference value, the first predicted difference value of the historical object is weighted, wherein the third reference value is a value less than 1; or Divide the first predicted difference value of the historical object by the fourth reference value, which is a positive number greater than 1.

15. An object recognition device, characterized in that, include: The recognition unit utilizes a trained object recognition network to perform multiple recognition processes on road images containing objects to be processed, obtaining multiple object recognition information. The object recognition information includes at least predicted keypoint information and category information obtained after predicting the object to be processed. The object recognition network is trained based on the difference information between predicted keypoint information and corresponding labeled keypoint information of historical objects in sample road images. The object recognition network includes N sub-recognition networks; each sub-recognition network contains a recognition layer; the recognition layer includes a detection head structure and a detection output structure, the detection head structure including a second max-pooling layer and a detection convolutional layer. The system includes a global processing layer, wherein the first and second detection convolutional layers in the detection convolutional layer have the same kernel size but different strides and output channels; the global processing layer includes a global max pooling layer, a first processing convolutional layer, and a second processing convolutional layer, which have the same kernel size and stride but different output channels; the output of the detection convolutional layer is multiplied by the output of the global processing layer to obtain a first result; the second max pooling layer is added to the first result to obtain a second result; the second result is input into the detection output structure. The determining unit is used to determine the object category, location information, and key points obtained after recognizing the object to be processed based on the multiple object recognition information.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 14.

17. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 14 via the computer program.

Images