A data processing system, object detection method and device

Abstract

The present application discloses a data processing system, an object detection method and an apparatus thereof, which are applied to the field of artificial intelligence. In the present application, the second feature map generation unit introduces shallow texture detail information of the original feature map (multiple first feature maps generated by the convolution processing unit) into the deep feature map (multiple second feature maps generated by the first feature map generation unit) to generate multiple third feature maps. The third feature map with shallow rich texture detail information is used as input data for target detection by the detection unit, which can improve the detection accuracy of subsequent object detection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing system, an object detection method and a device thereof. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras / camcorders and computers to obtain the data and information of the objects we need. Figuratively speaking, it is to install eyes (cameras or camcorders) and brains (algorithms) on computers to replace human eyes to identify, track, and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] The perception network can be a neural network model that processes and analyzes images and obtains processing results. At present, the perception network can complete more and more functions, for example, it can be image classification, 2D detection, semantic segmentation, key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable area detection, etc. In addition, the visual perception system has the characteristics of low cost, non-contact, small size, and large amount of information. With the continuous improvement of the accuracy of visual perception algorithms, it has become a key technology of many artificial intelligence systems today and has been more and more widely used, such as: advanced driving assistant system (ADAS) and autonomous driving system (ADS) on the road The recognition of dynamic obstacles (people or cars) and static objects (traffic lights, traffic signs or traffic cones), and the recognition of human masks and key points in the camera beauty function of terminal vision achieves a slimming effect.

[0004] Perception networks usually include feature pyramid networks (FPNs), which introduce a top-to-bottom network structure and a lateral connection branch derived from the original feature extraction network to fuse the corresponding resolution feature map of the original feature network with the deep feature map after upsampling. The top-to-bottom network structure introduced in FPN has a larger receptive field, but its detection accuracy for small objects is low. Summary of the invention

[0005] In a first aspect, the present application provides an object detection method, which is used in a first perception network, and the method includes:

[0006] Receiving an input first image, and performing convolution processing on the first image to generate a plurality of first feature maps;

[0007] It should be noted that the “performing convolution processing on the input image” here should not be understood as merely performing convolution processing on the input image. In some implementations, convolution processing, pooling operations, etc. may be performed on the input image.

[0008] It should be noted that the “convolution processing is performed on the first image to generate multiple first feature maps” here should not be understood as merely performing convolution processing on the first image multiple times, and each convolution processing can generate a first feature map, that is, it should not be understood as each first feature map is obtained based on the convolution processing on the first image, but, from a holistic perspective, the first image is the source of multiple first feature maps; in one implementation, the first image can be convolved to obtain a first feature map, and then the generated first feature map can be convolved to obtain another first feature map, and so on, to obtain multiple first feature maps.

[0009] It should be noted that a series of convolution processes may be performed on the input image. Specifically, in each convolution process, the first feature map obtained by the previous convolution process may be convolved to obtain a first feature map. In the above manner, multiple first feature maps may be obtained.

[0010] It should be noted that the multiple first feature maps may be feature maps with multi-scale resolutions, that is, the multiple first feature maps are not feature maps with the same resolution. In an optional implementation, the multiple first feature maps may constitute a feature pyramid.

[0011] Among them, an input image can be received, and a convolution process can be performed on the input image to generate multiple first feature maps with multi-scale resolutions; the convolution processing unit can perform a series of convolution processes on the input image to obtain feature maps at different scales (with different resolutions). The convolution processing unit can take many forms, such as visual geometry group (VGG), residual neural network (resnet), GoogLeNet's core structure (Inception-net), etc.

[0012] A plurality of second feature maps are generated according to the plurality of first feature maps; wherein the plurality of first feature maps include more texture details of the input image and / or position details in the input image than the plurality of second feature maps.

[0013] It should be noted that the “generating multiple second feature maps based on the multiple first feature maps” here should not be understood as the source of each second feature map in the multiple second feature maps is the multiple first feature maps; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on one or more first feature maps in the multiple first feature maps; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on one or more first feature maps in the multiple first feature maps, and other second feature maps except themselves; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on other second feature maps except themselves. At this time, since “other second feature maps except themselves are generated based on one or more first feature maps in the multiple first feature maps”, it can be understood as generating multiple second feature maps based on the multiple first feature maps.

[0014] It should be noted that the multiple second feature maps may be feature maps with multi-scale resolutions, that is, the multiple second feature maps are not feature maps with the same resolution. In an optional implementation, the multiple second feature maps may constitute a feature pyramid.

[0015] Among them, a convolution operation can be performed on the top-level feature map C4 among the multiple first feature maps generated by the convolution processing unit. Exemplarily, the number of channels of the top-level feature map C4 can be reduced to 256 by using void convolution and 1×1 convolution, as the top-level feature map P4 of the feature pyramid; the output result of the next layer feature map C3 of the top layer is horizontally linked and the number of channels is reduced to 256 by using 1×1 convolution, and then the output result is added to the feature map P4 channel by channel and pixel by pixel to obtain the feature map P3; and so on, from top to bottom, a first feature pyramid is constructed, and the first feature pyramid may include multiple second feature maps.

[0016] The texture details can be used to express the details of small targets and edge features. Compared with the second feature map, the first feature map includes more texture details, so that the detection accuracy of the detection results for small target detection is higher. The position details can be information expressing the position of objects in the image and the relative positions between objects.

[0017] Compared with the first feature map, multiple second feature maps can include more deep features. The deep features contain rich semantic information, which has a good effect on classification tasks. At the same time, the deep features have a large receptive field and can have a better detection effect on large targets. In one implementation, by introducing a top-down pathway to generate multiple second feature maps, the rich semantic information contained in the deep features can be naturally propagated downward, so that the second feature maps of each scale contain rich semantic information.

[0018] A plurality of third feature maps are generated according to the plurality of first feature maps and the plurality of second feature maps.

[0019] It should be noted that the phrase “generating multiple third feature maps based on the multiple first feature maps and the multiple second feature maps” here should not be understood as that the source of each third feature map in the multiple second feature maps is the multiple first feature maps and the multiple second feature maps, but rather, from a holistic perspective, the multiple first feature maps and the multiple second feature maps are the source of the multiple third feature maps; in one implementation, a portion of the multiple third feature maps are generated based on one or more first feature maps in the multiple first feature maps and one or more second feature maps in the multiple second feature maps; in one implementation, a portion of the multiple third feature maps are generated based on one or more first feature maps in the multiple first feature maps, one or more second feature maps in the multiple second feature maps, and other third feature maps except themselves; in one implementation, a portion of the multiple third feature maps are generated based on other third feature maps except themselves.

[0020] It should be noted that the multiple third feature maps may be feature maps with multi-scale resolutions, that is, the multiple third feature maps are not feature maps with the same resolution. In an optional implementation, the multiple third feature maps may constitute a feature pyramid.

[0021] According to at least one third feature map among the plurality of third feature maps, a first detection result of the object included in the image is output.

[0022] In one implementation, the object may be a person, an animal, a plant, an item, etc.

[0023] In one implementation, object detection may be performed on the image based on at least one third feature map among the multiple third feature maps, wherein the object detection is to identify the type of object included in the image, the location of the object, etc.

[0024] In an existing implementation, the second feature map generation unit (such as a feature pyramid network) propagates the rich semantic information contained in the deep features downward by introducing a top-down path, so that the second feature maps of each scale contain rich semantic information. At the same time, the deep features have a large receptive field, so that a better detection effect can be achieved for large targets. However, in the existing implementation, the more refined position detail information and texture detail information contained in the shallower feature maps are ignored, which has a great impact on the detection accuracy of medium and small targets. In an embodiment of the present application, the second feature map generation unit introduces the shallow texture detail information of the original feature map (multiple first feature maps generated by the convolution processing unit) into the deep feature map (multiple second feature maps generated by the first feature map generation unit) to generate multiple third feature maps, and the third feature map with shallow rich texture detail information is used as the input data for the detection unit to perform target detection, which can improve the detection accuracy of subsequent object detection.

[0025] It should be noted that this embodiment does not mean that the detection accuracy of objects in any image including small targets will be higher, but for a large number of samples, this embodiment can have a higher comprehensive detection accuracy.

[0026] It should be noted that the above-mentioned object detection method can be implemented by a data processing system, such as a trained perception network, wherein the perception network may include a convolution processing unit, a first feature map generation unit, a second feature map generation unit and a detection unit, wherein the convolution processing unit is connected to the first feature map generation unit and the second feature map generation unit respectively, the first feature map generation unit is connected to the second feature map generation unit, and the second feature map generation unit is connected to the detection unit, the convolution processing unit is used to receive an input image and perform convolution processing on the input image to generate a plurality of first feature maps; the first feature map generation unit is used to generate a plurality of second feature maps based on the plurality of first feature maps; wherein the plurality of first feature maps include more texture details of the input image and / or position details in the input image than the plurality of second feature maps; the second feature map generation unit is used to generate a plurality of third feature maps based on the plurality of first feature maps and the plurality of second feature maps; the detection unit is used to output a detection result of an object included in the image based on at least one third feature map among the plurality of third feature maps.

[0027] In one implementation, the perception network may include: a backbone network, a first feature pyramid network FPN, a second FPN and a head-end header, the backbone network is connected to the first FPN and the second FPN respectively, the first FPN is connected to the second FPN, and the second FPN is connected to the head (wherein the convolution processing unit is the backbone network, the first feature map generation unit is the first feature pyramid network FPN, the second feature map generation unit is the second feature pyramid network FPN, and the detection unit is the head-end head).

[0028] The backbone network can be used to receive an input image and perform convolution processing on the input image to generate multiple first feature maps with multi-scale resolutions; the backbone network can perform a series of convolution processing on the input image to obtain feature maps at different scales (with different resolutions). The backbone network can take many forms, such as visual geometry group (VGG), residual neural network (resnet), GoogLeNet's core structure (Inception-net), etc.

[0029] The first FPN can be used to generate a first feature pyramid based on the multiple first feature maps, wherein the first feature pyramid includes multiple second feature maps with multi-scale resolutions; wherein a convolution operation is performed on the top-level feature map C4 generated by the backbone network. Exemplarily, the number of channels of the top-level feature map C4 can be reduced to 256 using hole convolution and 1×1 convolution, as the top-level feature map P4 of the feature pyramid; the output result of the feature map C3 of the next layer of the top layer is horizontally linked and the number of channels is reduced to 256 using 1×1 convolution, and then the output result is added to the feature map P4 channel by channel and pixel by pixel to obtain the feature map P3; and so on, from top to bottom, to construct the first feature pyramid.

[0030] The second FPN may be used to generate a second feature pyramid according to the plurality of first feature maps and the plurality of second feature maps, wherein the second feature pyramid includes a plurality of third feature maps having multi-scale resolutions;

[0031] The head is used to detect the target object in the image according to at least one third feature map among the multiple third feature maps, and output a detection result.

[0032] In an embodiment of the present application, the second FPN introduces shallow and rich edge, texture and other detail information of the original feature map (multiple first feature maps generated by the backbone network) into the deep feature map (multiple second feature maps generated by the first FPN) to generate a second feature pyramid. The third feature map with shallow and rich edge, texture and other detail information is used as input data for the head for target detection, which can improve the detection accuracy of subsequent object detection.

[0033] In an optional implementation, the multiple first feature maps include a first target feature map, the multiple second feature maps include a second target feature map, the multiple third feature maps include a third target feature map and a fourth target feature map, and the resolution of the third target feature map is smaller than that of the fourth target feature map; generating multiple third feature maps according to the multiple first feature maps and the multiple second feature maps includes:

[0034] Downsampling the third target feature map to obtain a fifth target feature map, wherein the fifth target feature map has the same number of channels and resolution as the second target feature map; downsampling and convolution processing are performed on the first target feature map to obtain a sixth target feature map, wherein the sixth target feature map has the same number of channels and resolution as the second target feature map; channel superposition of the fifth target feature map, the second target feature map, and the sixth target feature map is performed to generate the fourth target feature map;

[0035] In the embodiment of the present application, the third target feature map can be downsampled and convolved to obtain a fifth target feature map, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the fifth target feature map the same as that of the second target feature map, and the purpose of convolution is to make the number of channels of the fifth target feature map the same as that of the second target feature map. In the above manner, the fifth target feature map has the same resolution and number of channels as the second target feature map, and then the fifth target feature map, the sixth target feature map and the second target feature map can be added channel by channel to obtain the fourth target feature map.

[0036] Or, downsample the third target feature map to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; downsample the first target feature map to obtain a sixth target feature map, and the sixth target feature map has the same resolution as the second target feature map; superimpose channels of the fifth target feature map, the second target feature map and the sixth target feature map and perform convolution processing to generate the fourth target feature map, and the fourth target feature map has the same number of channels as the second target feature map.

[0037] In the embodiment of the present application, the expression "channel superposition" can be understood as superimposing (for example, performing addition operations, etc.) corresponding elements (corresponding elements can be understood as elements located at the same position on the feature map) in the feature map of corresponding channels (i.e., channels with the same semantic information).

[0038] In an embodiment of the present application, the second target feature map may be a feature map including multiple channels, and the feature map corresponding to each channel may be a feature map including a type of semantic information.

[0039] In an embodiment of the present application, the third target feature map can be downsampled to obtain a fifth target feature map, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the fifth target feature map the same as that of the second target feature map. In the above manner, the fifth target feature map has the same resolution as the second target feature map, and then the fifth target feature map and the second target feature map can be added channel by channel; the first target feature map can be downsampled to obtain a sixth target feature map, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the sixth target feature map the same as that of the second target feature map. In the above manner, the fifth target feature map, the sixth target feature map and the second target feature map have the same resolution, and then the fifth target feature map, the sixth target feature map and the second target feature map can be added channel by channel, and then through convolution processing, the fourth target feature map obtained has the same number of channels as the second target feature map, wherein the above convolution processing can be a concatenation operation.

[0040] In this embodiment, a third target feature map and a fourth target feature map with different resolutions can be generated, and the resolution of the third target feature map is smaller than that of the fourth target feature map, wherein the fourth target feature map with a larger resolution is generated based on a first feature map among multiple first feature maps, a second feature map among multiple second feature maps, and the third target feature map. At this time, the multiple third feature maps generated by the second feature map generation unit retain the advantages of the feature pyramid network, which has a bottom-up (generating feature maps with larger resolutions in sequence from feature maps with small resolutions), and introduces the rich texture detail information and / or position detail information of the shallow neural network into the deep convolutional layer. The detection network uses the small target detection results of the multiple third feature maps generated in this way to achieve higher detection accuracy.

[0041] In one implementation, the method further includes:

[0042] Convolving at least one third feature map among the plurality of third feature maps through the first convolutional layer to obtain at least one fourth feature map;

[0043] and performing convolution on at least one third feature map among the plurality of third feature maps to obtain at least one fifth feature map; wherein the receptive field corresponding to the fourth feature map is larger than the receptive field corresponding to the fifth feature map;

[0044] Correspondingly, outputting a first detection result of the object included in the image according to at least one third feature map among the plurality of third feature maps includes:

[0045] A first detection result of the object included in the image is outputted according to the at least one fourth feature map and the at least one fifth feature map.

[0046] In this embodiment, the fourth feature map and the fifth feature map obtained after processing the third feature map have different receptive fields. Due to the feature maps with different receptive fields, targets of different sizes can be adaptively detected. For example, the third feature map can be processed by convolutional layers with different dilation rates, and the processing results obtained can include information about both large targets and small targets, so that both large and small targets can be detected in the subsequent target detection process.

[0047] In one implementation, the method further includes:

[0048] Processing the fourth feature map according to the first weight value to obtain a processed fourth feature map;

[0049] It should be noted that the first weight value may be used to perform a channel-based multiplication operation or other numerical processing on the fourth feature map, so that the elements in the fourth feature map are gained by a corresponding magnitude.

[0050] The fifth feature map is processed according to the second weight value to obtain a processed fifth feature map; it should be noted that the second weight value can be used to perform a channel-based multiplication operation or other numerical processing on the fifth feature map so that the elements in the fifth feature map are gained by a corresponding amount.

[0051] Wherein, when the to-be-detected object included in the fourth feature map is larger than the to-be-detected object included in the fifth feature map, the first weight value is larger than the second weight value;

[0052] Correspondingly, outputting a first detection result of the object included in the image according to at least one third feature map among the plurality of third feature maps includes:

[0053] Outputting a first detection result of the object included in the image according to the processed fourth feature map and the processed fifth feature map.

[0054] In this embodiment, if the object to be detected included in the fourth feature map is larger than the object to be detected included in the fifth feature map, then correspondingly, the first weight value is greater than the second weight value, and the gain obtained by the processed fourth feature map relative to the processed fifth feature map will be greater. Since the receptive field corresponding to the fourth feature map itself is larger than the receptive field corresponding to the fifth feature map, the feature map with a larger receptive field has more information about large targets, and the detection accuracy of large targets in target detection performed using it is also higher. In this embodiment, when there is a larger target in the image, the corresponding gain of the fourth feature map is also greater than that of the fifth feature map. When the detection unit performs target detection on the image based on the processed fourth feature map and the processed fifth feature map, the overall receptive field will be larger, and correspondingly, the detection accuracy will be higher.

[0055] In this embodiment, the intermediate feature extraction layer can learn the recognition rule of the weight value through training: for the feature map including a large target, the first weight value of the first convolution layer determined by it is larger, and the second weight value of the second convolution layer determined by it is smaller. For the feature map including a small target, the first weight value of the first convolution layer determined by it is smaller, and the second weight value of the second convolution layer determined by it is larger.

[0056] In one implementation, the method further includes:

[0057] Performing a dilated convolution process on at least one third feature map among the plurality of third feature maps;

[0058] According to the at least one third feature map after the hole convolution processing, the target object in the image is detected, and a first detection result is output.

[0059] In an embodiment of the present application, there may be a 3x3 convolution in the candidate region extraction network (RPN) to act as a sliding window. By moving the convolution kernel on at least one third feature map, the subsequent intermediate layer, category judgment, and border regression layer can be used to obtain whether there is a target in the border of each anchor point (Anchor) and the difference between the predicted border and the real border. By training the candidate region extraction network, a better border extraction result can be obtained. In this embodiment, the 3x3 sliding window convolution kernel is replaced with a 3x3 hole convolution kernel, and a hole convolution process is performed on at least one third feature map of the multiple third feature maps. According to the at least one third feature map after the hole convolution process, the target object in the image is detected, and the detection result is output. Without increasing the amount of calculation, the receptive field is increased and the missed detection of large targets and partially occluded targets is reduced.

[0060] In one implementation, the first detection result includes a first detection frame, and the method further includes:

[0061] Acquire a second detection result of the first image, where the second detection result is obtained by performing object detection on the first image through a second perception network, the object detection accuracy of the first perception network is higher than the object detection accuracy of the second perception network, and the second detection result includes a second detection box, and there is an intersection between an area where the second detection box is located and an area where the first detection box is located;

[0062] If the ratio of the area of ​​the intersection to the area of ​​the first detection frame is smaller than a preset value, the second detection result is updated so that the updated second detection result includes the first detection frame.

[0063] In the embodiment of the present application, if the ratio of the area of ​​the first intersection to the area of ​​the first detection frame is less than a preset value, it can be considered that the first detection frame is omitted in the second detection result, and the second detection result can be updated so that the updated second detection result includes the first detection frame. The timing characteristics are introduced into the model to assist in determining whether the suspected missed detection result is a true missed detection result, and the category of the missed detection result is determined, thereby improving the verification efficiency.

[0064] In one implementation, the second detection result includes multiple detection frames, and there is an intersection between the area where each detection frame in the multiple detection frames is located and the area where the first detection frame is located, and the multiple detection frames include the second detection frame, wherein among the areas of the intersections of the areas where each detection frame in the multiple detection frames is located and the area where the first detection frame is located, the area of ​​the intersection between the area where the second detection frame is located and the area where the first detection frame is located is the smallest.

[0065] In one implementation, the first image is an image frame in a video, the second image is an image frame in the video, and a frame spacing between the first image and the second image in the video is less than a preset value, and the method further includes:

[0066] Acquire a third detection result of the second image, where the third detection result includes a fourth detection frame and an object category corresponding to the fourth detection frame;

[0067] When a shape difference and a position difference between the fourth detection frame and the first detection frame are within a preset range, the first detection frame corresponds to the object category corresponding to the fourth detection frame.

[0068] In one implementation, the detection confidence of the fourth detection frame is greater than a preset threshold.

[0069] In a second aspect, the present application provides a data processing system, comprising: a convolution processing unit, a first feature map generation unit, a second feature map generation unit and a detection unit, wherein the convolution processing unit is connected to the first feature map generation unit and the second feature map generation unit respectively, the first feature map generation unit is connected to the second feature map generation unit, and the second feature map generation unit is connected to the detection unit;

[0070] The convolution processing unit is used to receive an input image and perform convolution processing on the input image to generate a plurality of first feature maps;

[0071] The first feature map generating unit is used to generate a plurality of second feature maps according to the plurality of first feature maps; wherein the plurality of first feature maps include more texture details of the input image and / or position details in the input image than the plurality of second feature maps;

[0072] The second feature map generating unit is used to generate a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps;

[0073] The detection unit is used to output a detection result of the object included in the image according to at least one third feature map among the multiple third feature maps.

[0074] For example, the data processing system may be a perceptual network system for implementing the functions of a perceptual network.

[0075] In an existing implementation, the second feature map generation unit (such as a feature pyramid network) propagates the rich semantic information contained in the deep features downward by introducing a top-down path, so that the second feature maps of each scale contain rich semantic information. At the same time, the deep features have a large receptive field, so that a better detection effect can be achieved for large targets. However, in the existing implementation, the more refined position detail information and texture detail information contained in the shallower feature maps are ignored, which has a great impact on the detection accuracy of medium and small targets. In an embodiment of the present application, the second feature map generation unit introduces the shallow texture detail information of the original feature map (multiple first feature maps generated by the convolution processing unit) into the deep feature map (multiple second feature maps generated by the first feature map generation unit) to generate multiple third feature maps, and the third feature map with shallow rich texture detail information is used as the input data for the detection unit to perform target detection, which can improve the detection accuracy of subsequent object detection.

[0076] It should be noted that this embodiment does not mean that the detection accuracy of objects in any image including small targets will be higher, but for a large number of samples, this embodiment can have a higher comprehensive detection accuracy.

[0077] Since the third aspect is a device corresponding to the first aspect, its various implementation methods, explanations and corresponding technical effects please refer to the description of the first aspect and will not be repeated here.

[0078] In a third aspect, the present application provides a perceptual network training method, the method comprising:

[0079] Get the pre-annotated detection box of the target object in the image;

[0080] Acquire a target detection frame corresponding to the image and the first perception network, wherein the target detection frame is used to identify the target object;

[0081] In one design, a detection result of the image may be obtained, where the detection result is obtained by performing object detection on the image through a first perception network, and the detection result includes a target detection box corresponding to the first object;

[0082] Iteratively training the first perception network according to a loss function to output a second perception network; wherein the loss function is related to an intersection over union (IoU) ratio between the pre-labeled detection box and the target detection box;

[0083] In one design, the first perception network may be iteratively trained according to a loss function to update parameters included in the first perception network to obtain a second perception network; wherein the loss function is related to an intersection over union (IoU) ratio between the pre-labeled detection box and the target detection box.

[0084] In one design, the second perceptual network may be output.

[0085] In the embodiment of the present application, the newly designed border regression loss function utilizes the IoU loss term with scale invariance, which is applied to the target detection measurement method, the loss term that considers the aspect ratio of the predicted border and the true border, and the loss term of the ratio of the distance between the center coordinate of the predicted border and the center coordinate of the true border to the distance between the lower right corner coordinate of the predicted border and the upper left corner coordinate of the true border. The IoU loss term naturally introduces a scale-invariant border prediction quality evaluation index. The loss term of the aspect ratio of the two borders measures the degree of fit of the shapes between the two borders. The third distance ratio measurement term is used to solve the problem that when IoU=0, the relative position relationship between the predicted border and the true border cannot be known and it is difficult to perform back propagation.

[0086] In one implementation, the preset loss function is also related to a shape difference between the target detection box and the pre-labeled detection box, wherein the shape difference is negatively correlated with an area of ​​the pre-labeled detection box.

[0087] In one implementation, the preset loss function is also related to the position difference between the target detection box and the pre-labeled detection box in the image, wherein the position difference is negatively correlated with the area of ​​the pre-labeled detection box; or the position difference is negatively correlated with the area of ​​the minimum circumscribed rectangle of the convex hull of the pre-labeled detection box and the target detection box.

[0088] In one implementation, the target detection box includes a first corner point and a first center point, the pre-labeled detection box includes a second corner point and a second center point, the first corner point and the second corner point are two endpoints of a rectangular diagonal, and the position difference is also positively correlated with the position difference between the first center point and the second center point in the image, and negatively correlated with the length of the first corner point and the second corner point.

[0089] In one implementation, the preset loss function includes a target loss term related to the position difference, and the target loss term changes as the position difference changes; wherein, when the position difference is greater than a preset value, the rate of change of the target loss term is greater than a first preset rate of change; and / or, when the position difference is less than a preset value, the rate of change of the target loss term is less than a second preset rate of change. When the position difference is greater than a preset value, the rate of change of the target loss term is greater than the first preset rate of change; and / or, when the position difference is less than a preset value, the rate of change of the target loss term is less than the second preset rate of change, which can achieve a fast convergence effect during the training process.

[0090] In a fourth aspect, the present application provides a perceptual network training device, the device comprising:

[0091] An acquisition module, configured to acquire a pre-labeled detection frame of a target object in an image; and to acquire a target detection frame corresponding to the image and the first perception network, wherein the target detection frame is used to identify the target object;

[0092] An iterative training module is used to iteratively train the first perception network according to a loss function to output a second perception network; wherein the loss function is related to the intersection over union (IoU) ratio between the pre-labeled detection box and the target detection box.

[0093] In one implementation, the preset loss function is also related to a shape difference between the target detection box and the pre-labeled detection box, wherein the shape difference is negatively correlated with an area of ​​the pre-labeled detection box.

[0094] In one implementation, the preset loss function is also related to the position difference between the target detection box and the pre-labeled detection box in the image, wherein the position difference is negatively correlated with the area of ​​the pre-labeled detection box; or the position difference is negatively correlated with the area of ​​the minimum circumscribed rectangle of the convex hull of the pre-labeled detection box and the target detection box.

[0095] In one implementation, the target detection box includes a first corner point and a first center point, the pre-labeled detection box includes a second corner point and a second center point, the first corner point and the second corner point are two endpoints of a rectangular diagonal, and the position difference is also positively correlated with the position difference between the first center point and the second center point in the image, and negatively correlated with the length of the first corner point and the second corner point.

[0096] In one implementation, the preset loss function includes a target loss term related to the position difference, and the target loss term changes as the position difference changes; wherein,

[0097] When the position difference is greater than a preset value, the change rate of the target loss item is greater than a first preset change rate; and / or, when the position difference is less than a preset value, the change rate of the target loss item is less than a second preset change rate.

[0098] Since the fourth aspect is a device corresponding to the second aspect, its various implementation methods, explanations and corresponding technical effects please refer to the description of the first aspect and will not be repeated here.

[0099] In a fifth aspect, an embodiment of the present application provides an object detection device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the second aspect and any optional method of the second aspect mentioned above.

[0100] In a sixth aspect, an embodiment of the present application provides an object detection device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the third aspect and any optional method of the third aspect mentioned above.

[0101] In a seventh aspect, an embodiment of the present invention further provides a perception network application system, which includes at least one processor, at least one memory, at least one communication interface, and at least one display device. The processor, memory, display device, and communication interface are connected through a communication bus and communicate with each other.

[0102] Communication interface, used to communicate with other devices or communication networks;

[0103] The memory is used to store application code for executing the above solution, and the execution is controlled by the processor. The processor is used to execute the application code stored in the memory;

[0104] The code stored in the memory may execute the object detection method provided above or the method for training the perception network provided in the above embodiment;

[0105] The display device is used to display the image to be identified, and the 2D, 3D, Mask, key points and other information of the object of interest in the image to be identified.

[0106] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned second aspect and any optional method thereof or the first aspect and any optional method thereof.

[0107] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the third aspect and any optional method thereof.

[0108] In the tenth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0109] In the eleventh aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the third aspect and any optional method thereof.

[0110] In a twelfth aspect, the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or, information. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or the training device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0111] The present application provides a data processing system, in which the second feature map generation unit introduces shallow texture detail information of the original feature map (multiple first feature maps generated by the convolution processing unit) into the deep feature map (multiple second feature maps generated by the first feature map generation unit) to generate multiple third feature maps. The third feature map with shallow rich texture detail information is used as input data for the detection unit to perform target detection, which can improve the detection accuracy of subsequent object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0113] Figure 2 This is an application scenario of the embodiment of the present application;

[0114] Figure 3 This is an application scenario of the embodiment of the present application;

[0115] Figure 4 This is an application scenario of the embodiment of the present application;

[0116] Figure 5 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0117] Figure 6 This is a schematic diagram of the structure of the convolutional neural network used in the embodiment of the present application;

[0118] Figure 7 This is a schematic diagram of the structure of the convolutional neural network used in the embodiment of the present application;

[0119] Figure 8 A hardware structure of a chip provided in an embodiment of the present application;

[0120] Fig. 9 A schematic diagram of the structure of a perception network provided in an embodiment of the present application;

[0121] Fig.10 The structure of a backbone network is shown in FIG.

[0122] Fig.11 It is a schematic diagram of the structure of a first FPN;

[0123] Fig.12a Schematic diagram of a second FPN structure;

[0124] Figure 12b Schematic diagram of a second FPN structure;

[0125] Fig.12c Schematic diagram of a second FPN structure;

[0126] Fig.12d Schematic diagram of a second FPN structure;

[0127] Fig.12e Schematic diagram of a second FPN structure;

[0128] Fig.13a This is a schematic diagram of the structure of a head;

[0129] Fig.13b This is a schematic diagram of the structure of a head;

[0130] Fig.14a A schematic diagram of the structure of a perception network provided in an embodiment of the present application;

[0131] Fig.14b A schematic diagram of a dilated convolution kernel provided in an embodiment of the present application;

[0132] Fig.14c It is a schematic diagram of the processing flow of an intermediate feature extraction;

[0133] Fig.15 A flowchart of an object detection method provided in an embodiment of the present application;

[0134] Fig.16 A flowchart of a perceptual network training method provided in an embodiment of the present application;

[0135] Fig.17 A flowchart of an object detection method provided in an embodiment of the present application;

[0136] Fig.18 A schematic diagram of a perceptual network training device provided in an embodiment of the present application;

[0137] Fig.19 A schematic diagram of an object detection device provided in an embodiment of the present application;

[0138] Fig. 20 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0139] Fig.21 It is a structural schematic diagram of a training device provided in an embodiment of the present application;

[0140] Fig. 22 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0141] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0142] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0143] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0144] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0145] (1) Infrastructure

[0146] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0147] (2) Data

[0148] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0149] (3) Data processing

[0150] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0151] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0152] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0153] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0154] (4) General ability

[0155] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0156] (5) Smart products and industry applications

[0157] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0158] The embodiments of the present application are mainly used in fields such as driving assistance, automatic driving, and mobile terminals that require the completion of multiple sensing tasks. Figure 2 and Figure 3 As shown, Figure 2 The application scenario of the embodiment of the present application is shown. The embodiment of the present application is in the module of automatic data annotation in the data processing platform, and the dotted line frames the location of the present invention. The system is an intelligent data platform for human-machine collaboration, which is used to build artificial intelligence capabilities with higher efficiency, faster training and stronger models. The automatic data annotation module is an intelligent annotation system framework to solve the high cost of manual annotation and the small number of manual annotation sets.

[0159] The product implementation form of the embodiment of the present application is a program code contained in the intelligent data storage system and deployed on the server hardware. The network element whose function is enhanced or modified due to this solution belongs to soft modification and is a relatively independent module. Figure 3Taking the application scenario shown as an example, the program code of the embodiment of the present application exists in the runtime training module of the intelligent data system. When the program is running, the program code of the embodiment of the present application runs on the host storage and acceleration hardware (GPU / FPGA / dedicated chip) memory of the server. The possible impact in the future is that the data reading module may have read data from a certain FTP, file, database, or memory before. After adopting the embodiment of the present application, it is only necessary to update the data source to the interface of the functional module involved in this solution.

[0160] Figure 3 The implementation form of the present invention in the server and platform software is given, wherein the label generation device and the automatic calibration device are newly added modules of the present invention on the basis of the existing platform software.

[0161] The following is a brief introduction to the two application scenarios of ADAS / ADS visual perception system and mobile phone beautification.

[0162] Application scenario 1: ADAS / ADS visual perception system

[0163] like Figure 4 As shown in the figure, in ADAS and ADS, multiple types of 2D target detection need to be performed in real time, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, and buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, and bicycles), traffic signs (traffic signs, guide signs, billboards, red traffic lights (TrafficLight_Red) / yellow traffic lights (TrafficLight_Yellow) / green traffic lights (TrafficLight_Green) / black traffic lights (TrafficLight_Black), and road signs). In addition, in order to accurately obtain the area occupied by dynamic obstacles in 3D space, it is also necessary to perform 3D estimation on the dynamic obstacles and output 3D boxes. In order to fuse with the laser radar data, it is necessary to obtain the mask of the dynamic obstacle, so as to filter out the laser point cloud that hits the dynamic obstacle; in order to accurately park, it is necessary to detect the four key points of the parking space at the same time; in order to perform composition positioning, it is necessary to detect the key points of the static target. Using the technical solution provided in the embodiment of the present application, all or part of the above functions can be completed in the perception network.

[0164] Application scenario 2: Mobile phone beauty function

[0165] In a mobile phone, the mask and key points of the human body are detected through the perception network provided in the embodiment of the present application, and the corresponding parts of the human body can be enlarged or reduced, such as performing waist-tightening and buttocks-beautifying operations, thereby outputting a beautified image.

[0166] Application scenario 3: Image classification scenario:

[0167] After acquiring the image to be classified, the object recognition device uses the object recognition method of the present application to obtain the category of the object in the image to be classified, and then the image to be classified can be classified according to the category of the object in the image to be classified. For photographers, many photos are taken every day, including animals, people, and plants. The method of the present application can quickly classify the photos according to the content in the photos, which can be divided into photos containing animals, photos containing people, and photos containing plants.

[0168] When there are a large number of images, manual classification is inefficient, and people are easily fatigued when dealing with the same thing for a long time, and the classification results will have large errors. However, the method of the present application can quickly classify images without errors.

[0169] Application scenario 4: Product classification:

[0170] After the object recognition device obtains the image of the commodity, the object recognition method of the present application is used to obtain the category of the commodity in the image of the commodity, and then the commodity is classified according to the category of the commodity. For a large shopping mall or supermarket with a wide variety of commodities, the object recognition method of the present application can quickly complete the classification of commodities, reducing time expenditure and labor costs.

[0171] The following describes the method provided by this application from the perspectives of model training and model application:

[0172] The method for training a perception network provided in an embodiment of the present application relates to computer vision processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, and symbolizes and formalizes the training data (such as the image or image block of the object and the category of the object in the present application) through intelligent information modeling, extraction, preprocessing, and training, and finally obtains a trained perception network; and, in an embodiment of the present application, inputs input data (such as the image of the object in the present application) into the trained perception network to obtain output data (such as obtaining 2D, 3D, Mask, key points, and other information of the object of interest in the image in the present application).

[0173] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0174] (1) Object detection: Using image processing and related methods such as machine learning and computer graphics, object detection can determine the category of image objects and determine the detection box used to locate the object.

[0175] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter. The perception network in this embodiment may include a convolutional neural network for performing convolution processing on an image or performing convolution processing on a feature map to generate a feature map.

[0176] (3) Back propagation algorithm

[0177] The convolutional neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial super-resolution model are updated by back-propagating the error loss information, so that the error loss converges. The back-propagation algorithm is a back-propagation movement dominated by error loss, which aims to obtain the optimal parameters of the super-resolution model, such as the weight matrix. In this embodiment, when training the perception network, the perception network can be updated based on the back-propagation algorithm.

[0178] Figure 5 is a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 5 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: an image to be identified or an image block or an image in the embodiment of the application.

[0179] When the execution device 120 preprocesses the input data, or when the computing module 111 of the execution device 120 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 120 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0180] Finally, the I / O interface 112 returns the processing result, such as at least one of the above-mentioned image or image block or 2D, 3D, Mask and key point information of the object of interest in the image, to the client device 140 for providing to the user.

[0181] Optionally, the client device 140 may be a control unit in an autonomous driving system or a functional algorithm module in a mobile phone terminal. For example, the functional algorithm module may be used to implement tasks related to perception.

[0182] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. Among them, the target model / rule can be the perception network described in the subsequent embodiments, and the result provided to the user can be the object detection result in the subsequent embodiments.

[0183] exist Figure 5 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific form can be display, sound, action and other specific methods. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0184] It is worth noting that Figure 5 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 5 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0185] like Figure 5 As shown, the perception network can be obtained by training according to the training device 120.

[0186] The perception network may include a deep neural network with a convolutional structure. The structure of the convolutional neural network used in the embodiment of the present application may be as follows: Figure 6 As shown. Figure 6 In the example, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 may obtain an image to be processed, and the obtained image to be processed is passed to the convolutional layer / pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result may be obtained. Figure 6 The internal layer structure of CNN 200 is introduced in detail.

[0187] Convolutional layer / pooling layer 220:

[0188] Convolutional Layer:

[0189] like Figure 6 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0190] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0191] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. Their role in image processing is equivalent to a filter that extracts specific information from an input image matrix. The convolution operator may essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix is ​​usually processed one pixel by one pixel (or two pixels by two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image.

[0192] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0193] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0194] Pooling layer:

[0195] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 6 The layers 221-226 illustrated in 220 may be a convolutional layer followed by a pooling layer, or multiple convolutional layers may be followed by one or more pooling layers.

[0196] Neural Network Layer 230:

[0197] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information because, as mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image.

[0198] It should be noted that if Figure 6 The convolutional neural network 210 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0199] The perception network of the embodiment of the present application may include a deep neural network with a convolutional structure, wherein the structure of the convolutional neural network may be as follows: Figure 7 As shown. Figure 7 In the embodiment of the present invention, the convolutional neural network (CNN) 200 may include an input layer 110, a convolutional layer / pooling layer 120 (wherein the pooling layer is optional), and a neural network layer 130. Figure 6 compared to, Figure 7 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 120 are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0200] See also Figure 8 , Figure 8 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application. Figure 8 As shown, the data processing system may include:

[0201] A convolution processing unit 801, a first feature map generating unit 802, a second feature map generating unit 803 and a detection unit 804, wherein the convolution processing unit 801 is respectively connected to the first feature map generating unit 802 and the second feature map generating unit 803, the first feature map generating unit 802 is connected to the second feature map generating unit 803, and the second feature map generating unit 803 is connected to the detection unit 804.

[0202] In one implementation, the data processing system can realize the function of a perception network, wherein the convolution processing unit 801 is a backbone network, the first feature map generation unit 802 and the second feature map generation unit 803 are feature pyramid networks, and the detection unit 804 is a head end.

[0203] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a perception network provided in an embodiment of the present application. Fig. 9 As shown, the perception network includes:

[0204] A backbone network 901, a first feature pyramid network FPN 902, a second FPN 903 and a head end header 904, wherein the backbone network 901 is connected to the first FPN 902 and the second FPN 903 respectively, the first FPN 901 is connected to the second FPN 902, and the second FPN 903 is connected to the head 904;

[0205] In the embodiment of the present application, the architecture of the perception network can be as follows Fig. 9 The architecture shown in the figure is mainly composed of a backbone network 901, a first FPN 902, a second FPN 903 and a head end header 904.

[0206] In one implementation, the convolution processing unit 801 is a backbone network, and the convolution processing unit 801 is used to receive an input image and perform convolution processing on the input image to generate multiple first feature maps.

[0207] It should be noted that the “convolution processing is performed on the input image” here should not be understood as merely performing convolution processing on the input image. In some implementations, convolution processing and other processing may be performed on the input image.

[0208] It should be noted that the “convolution processing is performed on the first image to generate multiple first feature maps” here should not be understood as merely performing convolution processing on the first image multiple times, and each convolution processing can generate a first feature map, that is, it should not be understood as each first feature map is obtained based on the convolution processing on the first image, but, from a holistic perspective, the first image is the source of multiple first feature maps; in one implementation, the first image can be convolved to obtain a first feature map, and then the generated first feature map can be convolved to obtain another first feature map, and so on, to obtain multiple first feature maps.

[0209] It should be noted that a series of convolution processes may be performed on the input image. Specifically, in each convolution process, the first feature map obtained by the previous convolution process may be convolved to obtain a first feature map. In the above manner, multiple first feature maps may be obtained.

[0210] It should be noted that the multiple first feature maps may be feature maps with multi-scale resolutions, that is, the multiple first feature maps are not feature maps with the same resolution. In an optional implementation, the multiple first feature maps may constitute a feature pyramid.

[0211] The convolution processing unit can be used to receive an input image and perform convolution processing on the input image to generate multiple first feature maps with multi-scale resolutions; the convolution processing unit can perform a series of convolution processing on the input image to obtain feature maps at different scales (with different resolutions). The convolution processing unit can take many forms, such as visual geometry group (VGG), residual neural network (resnet), GoogLeNet's core structure (Inception-net), etc.

[0212] In an embodiment of the present application, the convolution processing unit may be a backbone network, wherein the backbone network 901 is used to receive an input image and perform convolution processing on the input image to generate a plurality of first feature maps with multi-scale resolutions.

[0213] Reference Fig.10 , Fig.10 A schematic diagram of the structure of a backbone network provided in an embodiment of the present application is shown in FIG. Fig.10As shown in the figure, the backbone network is used to receive an input image, perform convolution processing on the input image, and output feature maps (feature map C1, feature map C2, feature map C3, feature map C4) with different resolutions corresponding to the image; that is to say, feature maps of different sizes corresponding to the image are output, and the backbone network completes the extraction of basic features to provide corresponding features for subsequent detection.

[0214] Specifically, the backbone network can perform a series of convolutions on the input image to obtain feature maps at different scales (with different resolutions). These feature maps will provide basic features for subsequent detection modules. The backbone network can take many forms, such as the visual geometry group (VGG), residual neural network (resnet), the core structure of GoogLeNet (Inception-net), etc.

[0215] The backbone network can perform convolution processing on the input image to generate several convolution feature maps of different scales. Each feature map is a H*W*C matrix, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map.

[0216] The backbone can adopt a variety of existing convolutional network frameworks, such as VGG16, Resnet50, Inception-Net, etc. The following takes Resnet18 as the backbone as an example.

[0217] Assume that the resolution of the input image is H*W*3 (height H, width W, number of channels is 3, that is, RBG three channels). The input image can be convolved through a convolution layer Res18-Conv1 of Resnet18 to generate Featuremap (feature map) C1. This feature map is downsampled twice relative to the input image, and the number of channels is expanded to 64, so the resolution of C1 is H / 4*W / 4*64. C1 can be convolved through Res18-Conv2 of Resnet18 to obtain Featuremap C2, the resolution of this feature map is consistent with C1; C2 continues to be convolved through Res18-Conv3 to generate Featuremap C3, this feature map is further downsampled relative to C2, the number of channels is doubled, and its resolution is H / 8*W / 8*128; finally, C3 is convolved through Res18-Conv4 to generate Featuremap C4, whose resolution is H / 16*W / 16*256.

[0218] It should be noted that the backbone network in the embodiment of the present application may also be referred to as a backbone network, which is not limited here.

[0219] It should be noted that Fig.10 The backbone network shown in the figure is only one implementation method and does not constitute a limitation to the present application.

[0220] The first feature map generating unit 802 is used to generate a plurality of second feature maps according to the plurality of first feature maps; wherein the plurality of first feature maps include more texture detail information and / or position detail information than the plurality of second feature maps.

[0221] It should be noted that the “generating multiple second feature maps based on the multiple first feature maps” here should not be understood as the source of each second feature map in the multiple second feature maps is the multiple first feature maps; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on one or more first feature maps in the multiple first feature maps; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on one or more first feature maps in the multiple first feature maps, and other second feature maps except themselves; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on other second feature maps except themselves. At this time, since “other second feature maps except themselves are generated based on one or more first feature maps in the multiple first feature maps”, it can be understood as generating multiple second feature maps based on the multiple first feature maps.

[0222] It should be noted that the multiple second feature maps may be feature maps with multi-scale resolutions, that is, the multiple second feature maps are not feature maps with the same resolution. In an optional implementation, the multiple second feature maps may constitute a feature pyramid.

[0223] Among them, a convolution operation can be performed on the top-level feature map C4 among the multiple first feature maps generated by the convolution processing unit. Exemplarily, the number of channels of the top-level feature map C4 can be reduced to 256 by using void convolution and 1×1 convolution, as the top-level feature map P4 of the feature pyramid; the output result of the next layer feature map C3 of the top layer is horizontally linked and the number of channels is reduced to 256 by using 1×1 convolution, and then the output result is added to the feature map P4 channel by channel and pixel by pixel to obtain the feature map P3; and so on, from top to bottom, a first feature pyramid is constructed, and the first feature pyramid may include multiple second feature maps.

[0224] It should be noted that the texture detail information here can be shallow detail information used to express small targets and edge features. Compared with the second feature map, the first feature map includes more texture detail information, so that its detection accuracy for small target detection is higher. Among them, the position details can be information expressing the position of objects in the image and the relative positions between objects.

[0225] It should be noted that compared with the first feature map, multiple second feature maps can include more deep features. The deep features contain rich semantic information, which has a good effect on classification tasks. At the same time, the deep features have a large receptive field and can have a better detection effect on large targets. In one implementation, by introducing a top-down pathway to generate multiple second feature maps, the rich semantic information contained in the deep features can be naturally propagated downward, so that the second feature maps of each scale contain rich semantic information.

[0226] In the embodiment of the present application, the first feature map generating unit 802 may be the first FPN 902 .

[0227] In an embodiment of the present application, the first FPN 902 is used to generate a first feature pyramid based on the multiple first feature maps, and the first feature pyramid includes multiple second feature maps with multi-scale resolutions. In an embodiment of the present application, the first FPN is connected to the backbone network, and the first FPN can perform convolution processing and merging processing on multiple feature maps of different resolutions generated by the backbone network to construct a first feature pyramid.

[0228] Reference Fig.11 , Fig.11 This is a structural diagram of a first FPN. The first FPN 902 can generate a first feature pyramid based on the multiple first feature maps. The first feature pyramid includes multiple second feature maps (feature map P2, feature map P3, feature map P4, feature map P5) with multi-scale resolution. Among them, the top-level feature map C4 generated by the backbone network 901 is subjected to a convolution operation. For example, the number of channels of the top-level feature map C4 can be reduced to 256 using a hole convolution and a 1×1 convolution, as the top-level feature map P4 of the feature pyramid; the output result of the feature map C3 of the next layer of the top layer is horizontally linked and the number of channels is reduced to 256 using a 1×1 convolution, and then the feature map P3 is obtained by adding the feature map P4 channel by channel and pixel by pixel; and so on, from top to bottom, a first feature pyramid ΦP={feature map P2, feature map P3, feature map P4, feature map P5} is constructed.

[0229] In the embodiment of the present application, in order to obtain a larger receptive field, the first feature pyramid may further include a feature map P5, which may be generated by directly performing a convolution operation on the feature map P4. The feature map in the middle of the first feature pyramid may introduce the rich semantic information contained in the deep features into each feature layer through a top-down structure, so that feature maps of different scales contain richer semantic information, which may provide better semantic information for small targets and improve the classification performance of small targets.

[0230] It should be noted that Fig.11 The first FPN shown in the figure is only one implementation method and does not constitute a limitation to the present application.

[0231] The second feature map generating unit 803 is used to generate a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps.

[0232] It should be noted that the phrase “generating multiple third feature maps based on the multiple first feature maps and the multiple second feature maps” here should not be understood as that the source of each third feature map in the multiple second feature maps is the multiple first feature maps and the multiple second feature maps, but rather, from a holistic perspective, the multiple first feature maps and the multiple second feature maps are the source of the multiple third feature maps; in one implementation, a portion of the multiple third feature maps are generated based on one or more first feature maps in the multiple first feature maps and one or more second feature maps in the multiple second feature maps; in one implementation, a portion of the multiple third feature maps are generated based on one or more first feature maps in the multiple first feature maps, one or more second feature maps in the multiple second feature maps, and other third feature maps except themselves; in one implementation, a portion of the multiple third feature maps are generated based on other third feature maps except themselves.

[0233] It should be noted that the multiple third feature maps may be feature maps with multi-scale resolutions, that is, the multiple third feature maps are not feature maps with the same resolution. In an optional implementation, the multiple third feature maps may constitute a feature pyramid.

[0234] In the embodiment of the present application, the second feature map generating unit 803 may be a second FPN 903 .

[0235] In the embodiment of the present application, the second FPN 903 is used to generate a second feature pyramid according to the multiple first feature maps and the multiple second feature maps, and the second feature pyramid includes multiple third feature maps with multi-scale resolutions.

[0236] Reference Fig.12a , Fig.12aThe structure of a second FPN is shown in FIG. 903 . The second FPN 903 can generate a second feature pyramid based on the multiple first feature maps generated by the backbone network 901 and the multiple second feature maps generated by the first FPN 902, wherein the second feature pyramid may include multiple third feature maps (e.g. Fig.12a , feature graph Q1, feature graph Q2, feature graph Q3 and feature graph Q4 shown in FIG.

[0237] In an embodiment of the present application, the second feature pyramid includes multiple third feature maps with multi-scale resolutions, wherein the lowest-level feature map (i.e., the feature map with the lowest resolution) among the multiple third feature maps can be generated based on a first feature map generated by a backbone network and a second feature map generated by a first FPN.

[0238] Specifically, in one embodiment, the multiple first feature maps include a first target feature map, the multiple second feature maps include a second target feature map, the multiple third feature maps include a third target feature map, the third target feature map is a feature map with the smallest resolution among the multiple third feature maps, and the second FPN is used to generate the third target feature map by the following steps:

[0239] The first target feature map is downsampled and convolved to obtain a fourth target feature map, wherein the fourth target feature map has the same number of channels and resolution as the second target feature map; and the fourth target feature map and the second target feature map are added channel by channel to generate the third target feature map.

[0240] Reference Figure 12b , the multiple first feature maps include a first target feature map, and the first target feature map can be Figure 12b The feature map C2 in the figure, the plurality of second feature maps include a second target feature map, and the second target feature map can be Figure 12b The feature map P3 in the figure, the plurality of third feature maps include a third target feature map, and the third target feature map can be Figure 12b Feature map Q1 in .

[0241] In the embodiment of the present application, the first target feature map can be downsampled and convolved to obtain a fourth target feature map, such as Figure 12b As shown in , the feature map C2 can be downsampled and convolved, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the fourth target feature map the same as that of the feature map P3, and the purpose of convolution is to make the number of channels of the fourth target feature map the same as that of the feature map P3. In the above manner, the fourth target feature map and the second target feature map have the same number of channels and resolution, and then the fourth target feature map and the second target feature map can be added channel by channel, as shown in Figure 12bAs shown in , the fourth target feature map and the second target feature map are added channel by channel to obtain the feature map Q1.

[0242] Specifically, in another embodiment, the multiple first feature maps include a first target feature map, the multiple second feature maps include a second target feature map, the multiple third feature maps include a third target feature map, the third target feature map is a feature map with the smallest resolution among the multiple third feature maps, and the second FPN is used to generate the third target feature map by the following steps:

[0243] The first target feature map is downsampled to obtain a fourth target feature map, and the fourth target feature map has the same resolution as the second target feature map; the fourth target feature map and the second target feature map are added channel by channel and convolved to generate the third target feature map, and the third target feature map has the same number of channels as the second target feature map.

[0244] Reference Fig.12c , the multiple first feature maps include a first target feature map, and the first target feature map can be Fig.12c The feature map C2 in the figure, the plurality of second feature maps include a second target feature map, and the second target feature map can be Fig.12c The feature map P3 in the figure, the plurality of third feature maps include a third target feature map, and the third target feature map can be Fig.12c Feature map Q1 in .

[0245] In the embodiment of the present application, the first target feature map can be downsampled to obtain a fourth target feature map, such as Fig.12c As shown in , the feature map C2 can be downsampled, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the fourth target feature map the same as that of the feature map P3. In the above manner, the fourth target feature map and the second target feature map have the same resolution, and then the fourth target feature map and the second target feature map can be added channel by channel, and then the convolution process is performed to make the obtained third target feature map have the same number of channels as the feature map P3, wherein the convolution process can be a concatenation operation.

[0246] In an embodiment of the present application, the second feature pyramid includes multiple third feature maps with multi-scale resolutions, wherein non-bottom-level feature maps among the multiple third feature maps (i.e., feature maps whose resolution is not the lowest) can be generated based on a first feature map generated by a backbone network, a second feature map generated by a first FPN, and a third feature map of an adjacent bottom level.

[0247] Specifically, in one embodiment, the multiple first feature maps include a first target feature map, the multiple second feature maps include a second target feature map, the multiple third feature maps include a third target feature map and a fourth target feature map, the resolution of the third target feature map is smaller than that of the fourth target feature map, and the second feature map generation unit is used to generate the fourth target feature map by the following steps:

[0248] The third target feature map is downsampled to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; the first target feature map is downsampled and convolved to obtain a sixth target feature map, and the sixth target feature map has the same number of channels and resolution as the second target feature map; the fifth target feature map, the second target feature map and the sixth target feature map are superimposed based on their respective channels to generate the fourth target feature map.

[0249] Reference Fig.12d , the multiple first feature maps include a first target feature map, and the first target feature map can be Fig.12d The feature map C3 in the figure, the plurality of second feature maps include a second target feature map, and the second target feature map can be Fig.12d The feature map P4 in the embodiment, the plurality of third feature maps include a third target feature map and a fourth target feature map, and the third target feature map can be Fig.12d Feature map Q2 in .

[0250] In the embodiment of the present application, the third target feature map can be downsampled to obtain a fifth target feature map, such as Fig.12d As shown in , the feature map Q1 can be downsampled, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the fifth target feature map the same as that of the feature map P4.

[0251] In the embodiment of the present application, the third target feature map can be downsampled and convolved to obtain a fifth target feature map, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the fifth target feature map the same as that of the second target feature map, and the purpose of convolution is to make the number of channels of the fifth target feature map the same as that of the second target feature map. In the above manner, the fifth target feature map has the same resolution and number of channels as the second target feature map, and then the fifth target feature map, the sixth target feature map and the second target feature map can be added channel by channel to obtain the fourth target feature map.

[0252] Specifically, in another embodiment, the third target feature map can be downsampled to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; the first target feature map is downsampled to obtain a sixth target feature map, and the sixth target feature map has the same resolution as the second target feature map; the fifth target feature map, the second target feature map and the sixth target feature map are superimposed based on their respective channels and convolved to generate the fourth target feature map, and the fourth target feature map has the same number of channels as the second target feature map.

[0253] Reference Fig.12e , the multiple first feature maps include a first target feature map, and the first target feature map can be Fig.12e The feature map C3 in the figure, the plurality of second feature maps include a second target feature map, and the second target feature map can be Fig.12e The feature map P4 in the embodiment, the plurality of third feature maps include a third target feature map and a fourth target feature map, and the third target feature map can be Fig.12e Feature map Q2 in .

[0254] In an embodiment of the present application, the third target feature map can be downsampled to obtain a fifth target feature map, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the fifth target feature map the same as that of the second target feature map. In the above manner, the fifth target feature map has the same resolution as the second target feature map, and then the fifth target feature map and the second target feature map can be added channel by channel; the first target feature map can be downsampled to obtain a sixth target feature map, wherein the purpose of downsampling is to make the resolution of the feature map of each channel in the sixth target feature map the same as that of the second target feature map. In the above manner, the fifth target feature map, the sixth target feature map and the second target feature map have the same resolution, and then the fifth target feature map, the sixth target feature map and the second target feature map can be added channel by channel, and then through convolution processing, the fourth target feature map obtained has the same number of channels as the second target feature map, wherein the above convolution processing can be a concatenation operation.

[0255] In this embodiment, the second feature map generation unit can generate a third target feature map and a fourth target feature map with different resolutions, wherein the resolution of the third target feature map is smaller than that of the fourth target feature map, wherein the fourth target feature map with a larger resolution is generated based on a first feature map in a plurality of first feature maps, a second feature map in a plurality of second feature maps, and the third target feature map. At this time, the multiple third feature maps generated by the second feature map generation unit retain the advantages of the feature pyramid network, which has a bottom-up (generating feature maps with larger resolutions in sequence from feature maps with small resolutions), and introduces the rich texture detail information and / or position detail information of the shallow neural network into the deep convolutional layer, and the detection network uses the small target detection results of the multiple third feature maps generated in this way to have higher detection accuracy.

[0256] Object detection tasks are different from image classification. In image classification tasks, the model only needs to answer the question of what is in the image. Therefore, image classification tasks have translation invariance, such as translation and scale. In object detection tasks, the model needs to know where the target is in the image and which category the target belongs to at the same time. In the actual visible light image obtained, the size and position of the target are constantly changing, and there is a problem of multi-scale target detection. The original one-stage and two-stage models are based on the image classification task. Directly migrate part of the image classification network for detection, and in order to obtain feature maps with sufficient expressive power and semantic information, the deeper features of the network are generally selected for subsequent processing. In order to obtain better expressive power and higher precision results, the current deep neural network models are expanding towards deeper and multi-branch topological structures. With the deepening of the network layer, the problem of network degradation also appears. The ResNet network structure with identity mapping and jump connection can solve the problem of network degradation well. Currently, in high-precision detection models, the number of network layers is on the order of 10 to 100, which allows the network to obtain better expressive capabilities. However, the problem with this is that the deeper the network layer, the larger the receptive field of the acquired features, which can cause small targets to be missed during detection.

[0257] Moreover, in traditional detection algorithms, only the features obtained from a certain layer are used, which will produce a fixed receptive field. Although anchors will be introduced in subsequent processing, by setting anchors of different sizes and aspect ratios and mapping them back to the original image through feature maps for processing, the multi-scale problem will inevitably exist. The feature pyramid network (FPN) can solve the multi-scale target detection problem well. Through the observation of deep convolutional neural networks, the deep network model itself has multi-level and multi-scale feature maps. The shallow feature map has a smaller receptive field, and the deep feature map has a larger receptive field. In this way, directly using such a feature map with a pyramid hierarchy can introduce multi-scale information, but there is a problem. Although the shallow feature map has a smaller receptive field, which is beneficial for detecting small targets, the shallow feature map contains less semantic information. It can be understood that the shallow features are not abstract enough and are difficult to be used to classify the detected targets. The feature pyramid network (FPN) adopts a clever structural design to solve the problem of shallow feature semantic information is not rich. For a basic deep convolutional neural network, there is a bottom-to-top forward calculation process to obtain hierarchical feature maps, and each scale feature has a scaling factor, and the feature map resolution gradually decreases. The Feature Pyramid Network (FPN) introduces a top-to-bottom network structure, gradually enlarging the feature map resolution, and introducing a lateral connection branch derived from the original feature extraction network to fuse the original feature network's corresponding resolution feature map with the upsampled deep feature map.

[0258] In an embodiment of the present application, considering integrating the shallow information generated by the backbone network into each feature layer again, a multi-scale feature layer design with bottom-up jump connections is designed. The shallow feature map contains very rich edge, texture and detail information. By introducing a jump connection between the original feature map and the multi-scale network layer from bottom to top, and fusing the downsampled original feature map with the feature map of the corresponding resolution in the second multi-scale feature layer from top to bottom that is laterally connected, this network layer improvement has a better detection effect on small targets and partially occluded targets, and can introduce features with rich semantic information and detail information in the multi-scale feature pyramid layer.

[0259] In an existing implementation, the second feature map generation unit (such as a feature pyramid network) propagates the rich semantic information contained in the deep features downward by introducing a top-down path, so that the second feature maps of each scale contain rich semantic information. At the same time, the deep features have a large receptive field, so that a better detection effect can be achieved for large targets. However, in the existing implementation, the more refined position detail information and texture detail information contained in the shallower feature maps are ignored, which has a great impact on the detection accuracy of medium and small targets. In an embodiment of the present application, the second feature map generation unit introduces the shallow texture detail information of the original feature map (multiple first feature maps generated by the convolution processing unit) into the deep feature map (multiple second feature maps generated by the first feature map generation unit) to generate multiple third feature maps, and the third feature map with shallow rich texture detail information is used as the input data for the detection unit to perform target detection, which can improve the detection accuracy of subsequent object detection.

[0260] It should be noted that this embodiment does not mean that the detection accuracy of objects in any image including small targets will be higher, but for a large number of samples, this embodiment can have a higher comprehensive detection accuracy.

[0261] The detection unit 804 is used to perform target detection on the image according to at least one third feature map among the multiple third feature maps, and output a detection result.

[0262] In the embodiment of the present application, the detection unit 804 may be a head end.

[0263] In the embodiment of the present application, the head is used to detect the target object in the image according to at least one third feature map among the multiple third feature maps, and output the detection result.

[0264] In the embodiment of the present application, the perception network may include one or more heads, such as Fig.13a As shown, each parallel head end is used to detect a task object in a task according to the third feature map output by the second FPN, and output a 2D box of the area where the task object is located and the confidence corresponding to each 2D box; wherein each parallel head can complete the detection of different task objects; wherein the task object is an object that needs to be detected in the task; the higher the confidence, the greater the probability that the object corresponding to the task is in the 2D box corresponding to the confidence.

[0265] In the embodiment of the present application, different heads can complete different 2D detection tasks. For example, one head among multiple heads can complete car detection, and output the 2D box and confidence of Car / Truck / Bus; head1 among multiple heads can complete person detection, and output the 2D box and confidence of Pedestrian / Cyclist / Tricyle; head among multiple heads can complete traffic light detection, and output the 2D box and confidence of Red_Trafficligh / Green_Trafficlight / Yellow_TrafficLight / Black_TrafficLight.

[0266] In an embodiment of the present application, the perception network may include multiple serial heads; the serial heads are connected to a parallel head; it should be emphasized here that, in fact, the serial head is not necessary, and for scenes that only need to detect 2D boxes, there is no need to include a serial head.

[0267] Among them, the serial head can be used to: use the 2D frame of the task object of the task provided by the connected parallel head, extract the features of the area where the 2D frame is located on one or more feature maps on the second FPN, and predict the 3D information, Mask information or Keypoint information of the task object of the task according to the features of the area where the 2D frame is located. The serial head can be optionally connected in series behind the parallel head, and complete the 3D / Mask / Keypoint detection of the object inside the 2D frame on the basis of detecting the 2D frame of the task. For example, the serial 3D_head0 completes the estimation of the vehicle's orientation, center of mass, length, width and height, and outputs the vehicle's 3D frame; the serial Mask_head0 predicts the vehicle's fine mask, thereby segmenting the vehicle; and the serial Keypont_head0 completes the estimation of the vehicle's key points. The serial head is not necessary. Some tasks do not require 3D / Mask / Keypoint detection, so there is no need to connect the serial head in series. For example, the detection of traffic lights only requires the detection of 2D frames, so there is no need to connect the serial head in series. In addition, some tasks can choose to concatenate one or more serial heads according to the specific requirements of the task. For example, for parking lot detection, in addition to the 2D box, the key points of the parking space are also required. Therefore, in this task, only one serial Keypoint_head is needed, and 3D and Mask heads are not required.

[0268] In the embodiment of the present application, the header is connected to the FPN. The header can complete the detection of the 2D box of a task according to the feature map provided by the FPN, and output the 2D box of the object of the task and the corresponding confidence, etc. The following describes a schematic diagram of the structure of the header. Fig.13b , Fig.13b It is a kind of header indication, such as Fig.13b As shown in the figure, the head includes three modules: Region Proposal Network (RPN), ROI-ALIGN and RCNN.

[0269] Among them, the RPN module can be used to predict the area where the task object is located on one or more third feature maps provided by the second FPN, and output a candidate 2D box that matches the area; or it can be understood that the RPN predicts the area where the task object may exist on one or more horizontal maps output by the FPN, and gives the boxes of these areas, which are called candidate areas (Proposal). For example, when the head is responsible for detecting a car, its RPN layer predicts a candidate box where a car may exist; when the head is responsible for detecting a person, its RPN layer predicts a candidate box where a person may exist. Of course, these Proposals are inaccurate. On the one hand, they do not necessarily contain the object of the task, and on the other hand, these boxes are not compact.

[0270] The 2D candidate region prediction process can be implemented by the head's RPN module, which predicts the area where the task object may exist based on the feature map provided by FPN, and gives the candidate boxes of these areas (also called candidate areas, Proposal). In this embodiment, if the head is responsible for detecting cars, its RPN layer predicts the candidate boxes where cars may exist.

[0271] The RPN layer can generate the feature map RPN Hidden on the third feature map provided by the second FPN through a convolution of, for example, 3*3. The subsequent RPN layer in the head will predict Proposals from RPN Hidden. Specifically, the RPN layer in the head predicts the coordinates and confidence of the Proposals at each position in RPN Hidden through a 1*1 convolution. The higher this confidence, the greater the probability that the object of this task exists in this Proposal. For example, the larger the score of a Proposal in the head, the greater the probability that it contains a vehicle. The Proposals predicted by each RPN layer need to go through the Proposal merging module to remove redundant Proposals according to the overlapping degree between the Proposals (this process can adopt but is not limited to the NMS algorithm), and select the N (N < K) Proposals with the largest scores from the remaining K Proposals as the candidate regions where the object may exist. These Proposals are inaccurate. On the one hand, they may not contain the object of this task, and on the other hand, these bounding boxes are not tight. Therefore, the RPN module is only a rough detection process and requires the subsequent RCNN module for refinement. When the RPN module regresses the coordinates of the Proposal, it does not directly regress the absolute values of the coordinates, but regresses the coordinates relative to the Anchor. The higher the matching degree between these Anchors and the actual object, the greater the probability that the RPN can detect the object.

[0272] The ROI-ALIGN module is used to extract the features of the region where the candidate 2D bounding box is located from a feature map provided by the FPN according to the region predicted by the RPN module; that is to say, the ROI-ALIGN module mainly extracts the features of the region where each Proposal is located on a certain feature map according to the Proposals provided by the RPN module, and resizes them to a fixed size to obtain the features of each Proposal. It can be understood that the ROI-ALIGN module can use but is not limited to feature extraction methods such as ROI-POOLING (Region of Interest Pooling) / ROI-ALIGN (Region of Interest Alignment) / PS-ROIPOOLING (Position-Sensitive Region of Interest Pooling) / PS-ROIALIGN (Position-Sensitive Region of Interest Alignment).

[0273] The RCNN module is used to perform convolution processing on the features of the region where the candidate 2D frame is located through a neural network to obtain the confidence that the candidate 2D frame belongs to each object category; the coordinates of the 2D frame of the candidate region are adjusted through a neural network so that the adjusted 2D candidate frame is more closely matched to the shape of the actual object than the candidate 2D frame, and the adjusted 2D candidate frame with a confidence greater than a preset threshold is selected as the 2D frame of the region. In other words, the RCNN module mainly refines the features of each Proposal proposed by the ROI-ALIGN module to obtain the confidence that each Proposal belongs to each category (for example, for the task of the car, four scores of Background / Car / Truck / Bus will be given), and at the same time adjusts the coordinates of the Proposal's 2D frame to output a more compact 2D frame. These 2D frames are merged through nonmaximum suppression (NMS) and output as the final 2D frame.

[0274] 2D candidate region subdivision is mainly composed of Fig.13b The RCNN module of the head in the implementation further regresses more compact 2D box coordinates based on the features of each Proposal extracted by the ROI-ALIGN module, and at the same time classifies the Proposal and outputs the confidence that it belongs to each category. There are many implementable forms of RCNN. The feature size output by the ROI-ALIGN module can be N*14*14*256 (Feature of proposals). In the RCNN module, it is first processed by the convolution module 4 (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512. Then it is processed by a Global Avg Pool (average pooling layer) to average the 7*7 features in each channel of the input features to obtain N*512 features, where each 1*512-dimensional feature vector represents the features of each Proposal. Next, two fully connected layers FC are used to regress the precise coordinates of the box (output N*4 vectors, where the four values ​​represent the x / y coordinates of the center point of the box, the width and height of the box), and the confidence of the box category (in head0, the score of whether the box is background / car / truck / bus needs to be given). Finally, the box merging operation is used to select several boxes with the largest scores, and the NMS operation is used to remove duplicate boxes, so as to obtain a compact box output.

[0275] In some practical application scenarios, the perception network can also include other heads, which can further perform 3D / Mask / Keypoint detection based on the detection of 2D boxes. For example, taking 3D as an example, the ROI-ALIGN module extracts the features of the area where each 2D box is located on the feature map output by FPN based on the accurate 2D box provided by the head. Assuming that the number of 2D boxes is M, the feature size output by the ROI-ALIGN module is M*14*14*256, which is first processed by Resnet18's Res18-Conv5, and the output feature size is N*7*7*512, and then processed by a Global Avg Pool (average pooling layer), averaging the 7*7 features of each channel in the input features to obtain M*512 features, where each 1*512-dimensional feature vector represents the feature of each 2D box. Next, three fully connected layers FC are used to regress the orientation angle (orientation, M*1 vector), centroid coordinates (centroid, M*2 vector, these two values ​​represent the x / y coordinates of the centroid) and length, width and height (dimention) of the object in the frame.

[0276] It should be noted that Fig.13a and Fig.13b The header shown in is only one implementation method and does not constitute a limitation of the present application.

[0277] In an embodiment of the present application, the perception network may further include: a dilated convolution layer, used to perform dilated convolution processing on at least one third feature map among the multiple third feature maps; correspondingly, the head is specifically used to detect the target object in the image according to the at least one third feature map after the dilated convolution processing, and output the detection result.

[0278] Reference Fig.14a and Fig.14b , Fig.14a A schematic diagram of the structure of a perception network provided in an embodiment of the present application is provided. Fig.14bA schematic diagram of a dilated convolution kernel provided for an embodiment of the present application. In an embodiment of the present application, there will be a 3x3 convolution in the candidate region extraction network (RPN) that acts as a sliding window. By moving the convolution kernel on at least one third feature map, the subsequent intermediate layer, category judgment, and border regression layer can be used to obtain whether there is a target in the border of each anchor point (Anchor) and the difference between the predicted border and the real border. By training the candidate region extraction network, a better border extraction result can be obtained. In this embodiment, the 3x3 sliding window convolution kernel is replaced with a 3x3 dilated convolution kernel, and at least one of the multiple third feature maps is subjected to dilated convolution processing. According to the at least one third feature map after the dilated convolution processing, the target object in the image is detected, and the detection result is output. Without increasing the amount of calculation, the receptive field is increased, and the missed detection of large targets and partially occluded targets is reduced. Among them, assuming that the original ordinary convolution kernel space size is k×k, a new parameter d is introduced, and (d-1) spaces are stuffed into the original convolution kernel, then the new convolution kernel size is:

[0279] n=k+(k-1)×(d-1)

[0280] For example, for a 3x3 convolution kernel, set d=2, and the receptive field of the new convolution kernel is 5x5.

[0281] The existing candidate region extraction network (RPN) performs subsequent processing by setting a 3x3 ordinary convolution kernel as a sliding window to slide on the feature map. In this embodiment, the 3x3 ordinary convolution kernel is replaced with a 3x3 hole convolution kernel to improve the network's correction of missed detection of large targets and occluded targets. In this embodiment, after improving the feature extraction network and the candidate region extraction network (RPN), the final annotation model obtained can obtain good detection effects, and after replacing the ordinary convolution with the 3x3 hole convolution, a larger receptive field can be obtained without increasing the amount of calculation. At the same time, due to the larger receptive field, better contextual information can be obtained, reducing the occurrence of background misjudgment as foreground.

[0282] The present application provides a data processing system, characterized in that the data processing system includes: a convolution processing unit, a first feature map generation unit, a second feature map generation unit and a detection unit, the convolution processing unit is connected to the first feature map generation unit and the second feature map generation unit respectively, the first feature map generation unit is connected to the second feature map generation unit, and the second feature map generation unit is connected to the detection unit; the convolution processing unit is used to receive an input image and perform convolution processing on the input image to generate a plurality of first feature maps; the first feature map generation unit is used to generate a plurality of second feature maps based on the plurality of first feature maps; wherein the plurality of first feature maps include more texture details of the input image and / or position details in the input image than the plurality of second feature maps; the second feature map generation unit is used to generate a plurality of third feature maps based on the plurality of first feature maps and the plurality of second feature maps; the detection unit is used to output a detection result of an object included in the image based on at least one third feature map among the plurality of third feature maps. In an embodiment of the present application, the second feature map generation unit introduces shallow texture detail information of the original feature map (multiple first feature maps generated by the convolution processing unit) into the deep feature map (multiple second feature maps generated by the first feature map generation unit) to generate multiple third feature maps. The third feature map with shallow rich texture detail information is used as input data for the detection unit to perform target detection, which can improve the detection accuracy of subsequent object detection.

[0283] Optionally, the data processing system in the embodiment of the present application may further include:

[0284] The intermediate feature extraction layer is used to convolve at least one third feature map among the multiple third feature maps to obtain at least one fourth feature map; and to convolve at least one third feature map among the multiple third feature maps to obtain at least one fifth feature map; wherein the receptive field corresponding to the fourth feature map is larger than the receptive field corresponding to the fifth feature map; the detection unit is specifically used to output the detection result of the object included in the image according to the at least one fourth feature map and the at least one fifth feature map.

[0285] Reference Fig.14c , Fig.14c This is a schematic diagram of the processing flow of intermediate feature extraction, such as Fig.14c As shown in , the third feature map can be convolved using convolution layers with different dilation rates to obtain corresponding feature maps (the number of channels of each feature map is c), and then concatenated to obtain feature Figure 4(The number of channels is 3c), and then the global Info descriptor is obtained through global average pooling and arithmetic balance processing, and nonlinear features are given through the first fully connected layer. After processing through the second fully connected layer and the sigmoid function, each weight value is limited to the range of 0-1, and then the weight value and the corresponding feature map are multiplied channel by channel to obtain the processed feature map.

[0286] In this embodiment, if the object to be detected included in the fourth feature map is larger than the object to be detected included in the fifth feature map, then correspondingly, the first weight value is greater than the second weight value, and the gain obtained by the processed fourth feature map relative to the processed fifth feature map will be greater. Since the receptive field corresponding to the fourth feature map itself is larger than the receptive field corresponding to the fifth feature map, the feature map with a larger receptive field has more information about large targets, and the detection accuracy of large targets in target detection performed using it is also higher. In this embodiment, when there is a larger target in the image, the corresponding gain of the fourth feature map is also greater than that of the fifth feature map. When the detection unit performs target detection on the image based on the processed fourth feature map and the processed fifth feature map, the overall receptive field will be larger, and correspondingly, the detection accuracy will be higher.

[0287] In this embodiment, the intermediate feature extraction layer can learn the recognition rule of the weight value through training: for the feature map including a large target, the first weight value of the first convolution layer determined by it is larger, and the second weight value of the second convolution layer determined by it is smaller. For the feature map including a small target, the first weight value of the first convolution layer determined by it is smaller, and the second weight value of the second convolution layer determined by it is larger.

[0288] Reference Fig.15 , Fig.15 The process diagram of an object detection method provided in an embodiment of the present application is as follows: Fig.15 As shown in , the object detection method includes:

[0289] 1501. Receive a first input image.

[0290] In the embodiment of the present application, when object detection needs to be performed on the first image, an input first image may be received.

[0291] 1502. Perform object detection on the first image through a first perception network to obtain a first detection result, where the first detection result includes a first detection box.

[0292] In an embodiment of the present application, after receiving an input first image, object detection can be performed on the first image through a first perception network to obtain a first detection result, where the first detection result includes a first detection box, wherein the first detection box can represent the pixel position of a detected object in the first image.

[0293] 1503. Perform object detection on the first image through a second perception network to obtain a second detection result, where the second detection result includes a second detection box, and there is a first intersection between the second detection box and the first detection box.

[0294] In an embodiment of the present application, object detection may be performed on the first image through a second perception network to obtain a second detection result, where the second detection result includes a second detection box, wherein the first detection box may represent the pixel position of a detected object in the first image.

[0295] In the embodiment of the present application, there is a first intersection between the second detection frame and the first detection frame, that is, on the first image, the pixel position where the second detection frame is located overlaps with the pixel position where the first detection frame is located.

[0296] 1504. If the ratio of the area of ​​the intersection to the area of ​​the first detection frame is smaller than a preset value, updating the second detection result so that the updated second detection result includes the first detection frame.

[0297] In an embodiment of the present application, if the ratio of the area of ​​the first intersection to the area of ​​the first detection frame is less than a preset value, it can be considered that the first detection frame is omitted in the second detection result, and the second detection result can be updated so that the updated second detection result includes the first detection frame.

[0298] In the embodiment of the present application, the second detection result may further include a third detection frame, and there is a second intersection between the third detection frame and the first detection frame, and the area of ​​the second intersection is smaller than the area of ​​the first intersection.

[0299] In an embodiment of the present application, there may be multiple detection frames in the second detection result that have intersections with the first detection frame, and among them, the size of the intersection between the second detection frame and the first detection frame is the largest, that is, the size of the intersection with the first detection frame is the largest. If the ratio of the area of ​​the intersection between the second detection frame and the first detection frame to the area of ​​the first detection frame is smaller than a preset value, then the ratio of the area of ​​the intersection between the remaining detection frames that have intersections with the first detection frame and the first detection frame to the area of ​​the first detection frame is also smaller than the preset value.

[0300] In an embodiment of the present application, for the same image, the object detection accuracy of the first perception network is higher than the object detection accuracy of the second perception network, and the object detection accuracy is related to at least one of the following features: the shape or position of the detection frame, or the category of the object corresponding to the detection frame.

[0301] That is, in the embodiment of the present application, the object detection accuracy of the first perception network is higher than the object detection accuracy of the second perception network, that is, the detection results of the first perception network can be used to update the detection results of the second perception network.

[0302] Optionally, a first input image can be received and convolution processing can be performed on the first image to generate multiple first feature maps; multiple second feature maps can be generated based on the multiple first feature maps; wherein the multiple first feature maps include more texture detail information and / or position detail information than the multiple second feature maps; multiple third feature maps can be generated based on the multiple first feature maps and the multiple second feature maps; target detection can be performed on the first image based on at least one third feature map among the multiple third feature maps, and a first detection result can be output.

[0303] Optionally, the plurality of second feature maps include more semantic information than the plurality of first feature maps.

[0304] Optionally, the multiple first feature maps include a first target feature map, the multiple second feature maps include a second target feature map, the multiple third feature maps include a third target feature map and a fourth target feature map, and the resolution of the third target feature map is smaller than that of the fourth target feature map; the third target feature map is downsampled to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; the first target feature map is downsampled and convolved to obtain a sixth target feature map, and the sixth target feature map has the same number of channels and resolution as the second target feature map; the fifth target feature map and the second target feature map are convolved to obtain a sixth target feature map. and the sixth target feature map based on their respective channels to generate the fourth target feature map; or, down-sample the third target feature map to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; down-sample the first target feature map to obtain a sixth target feature map, and the sixth target feature map has the same resolution as the second target feature map; superimpose the fifth target feature map, the second target feature map and the sixth target feature map based on their respective channels and perform convolution processing to generate the fourth target feature map, and the fourth target feature map has the same number of channels as the second target feature map.

[0305] Optionally, at least one of the multiple third feature maps can be convolved through a first convolutional layer to obtain at least one fourth feature map; at least one of the multiple third feature maps can be convolved through a second convolutional layer to obtain at least one fifth feature map; wherein the receptive field corresponding to the fourth feature map is larger than the receptive field corresponding to the fifth feature map; according to the fourth feature map and the fifth feature map, target detection is performed on the first image, and a first detection result is output.

[0306] Optionally, determine a first weight value corresponding to the fourth feature map and a second weight value corresponding to the fifth feature map; process the fourth feature map according to the first weight value to obtain a processed fourth feature map; process the fifth feature map according to the second weight value to obtain a processed fifth feature map; wherein, in the case where the object to be detected included in the fourth feature map is larger than the object to be detected included in the fifth feature map, the first weight value is greater than the second weight value; perform target detection on the first image according to the processed fourth feature map and the processed fifth feature map, and output a first detection result.

[0307] Optionally, the first detection result includes a first detection frame, and a second detection result of the first image may be obtained, the second detection result being obtained by performing object detection on the first image through a second perception network, the object detection accuracy of the first perception network is higher than that of the second perception network, the second detection result includes a second detection frame, and there is an intersection between an area where the second detection frame is located and an area where the first detection frame is located;

[0308] If the ratio of the area of ​​the intersection to the area of ​​the first detection frame is smaller than a preset value, the second detection result is updated so that the updated second detection result includes the first detection frame.

[0309] Optionally, the second detection result includes multiple detection frames, and there is an intersection between the area where each detection frame in the multiple detection frames is located and the area where the first detection frame is located, and the multiple detection frames include the second detection frame, wherein among the areas of the intersections of the areas where each detection frame in the multiple detection frames is located and the areas where the first detection frame is located, the area of ​​the intersection between the areas where the second detection frame is located and the areas where the first detection frame is located is the smallest.

[0310] Optionally, the first image is an image frame in a video, the second image is an image frame in the video, the frame spacing between the first image and the second image in the video is less than a preset value, and a third detection result of the second image is obtained, the third detection result including a fourth detection box and an object category corresponding to the fourth detection box; when the shape difference and position difference between the fourth detection box and the first detection box are within a preset range, the first detection box corresponds to the object category corresponding to the fourth detection box.

[0311] Optionally, the detection confidence of the fourth detection frame is greater than a preset threshold.

[0312] In the embodiment of the present application, the first image is an image frame in a video, the second image is an image frame in a video, the frame spacing between the first image and the second image in the video is less than a preset value, and a third detection result of the second image can also be obtained, the third detection result is obtained by performing object detection on the second image through the first perception network or the second perception network, and the third detection result includes a fourth detection frame and an object category corresponding to the fourth detection frame; if the shape difference and position difference between the fourth detection frame and the first detection frame are within a preset range, it is determined that the first detection frame corresponds to the object category corresponding to the fourth detection frame. Optionally, the detection confidence of the fourth detection frame is greater than a preset threshold.

[0313] In an embodiment of the present application, if the first image is obtained by extracting frames from a video, a timing detection algorithm may be considered. For missed targets detected by the missed detection algorithm, several frames are selected from the frames before and after the current image. By comparing the areas near the center of the missed target in the previous and next frames to see if there are targets with similar areas, aspect ratios, and center coordinates to the missed target, a specific number of similar target frames are selected, and then the similar target frames are compared with other target frames in the suspected missed image to see if there are similar target frames. Target frames similar to other target frames are removed from the detected similar target frames. The most similar target frame in each of the previous and next frames is obtained based on the content similarity and feature similarity algorithms, and the most similar target frame of the suspected missed target frame is obtained. If the confidence of the most similar target frame is higher than a certain threshold, the category of the missed target can be determined. If it is lower than a certain threshold, the suspected missed target is compared with the category of the most similar target frame. If the categories are the same, it is judged as missed. If the categories are different, manual verification can be performed.

[0314] The embodiment of the present application provides an object detection method, the method comprising: receiving an input first image; performing object detection on the first image through a first perception network to obtain a first detection result, the first detection result including a first detection frame; performing object detection on the first image through a second perception network to obtain a second detection result, the second detection result including a second detection frame, and a first intersection exists between the second detection frame and the first detection frame; if the ratio of the area of ​​the first intersection to the area of ​​the first detection frame is less than a preset value, then updating the two detection results, and the updated second detection result includes the first detection frame. In the above manner, the timing characteristics are introduced into the model to assist in determining whether the suspected missed detection result is a true missed detection result, and the category of the missed detection result is determined, thereby improving the verification efficiency.

[0315] Reference Fig.16 , Fig.16 A process diagram of a perception network training method provided for an embodiment of the present application, the perception network training method can be used to train the perception network in the above-mentioned embodiment. It should be noted that the first perception network in this embodiment can be the initial network of the perception network in the above-mentioned embodiment, and the second perception network obtained by training the first perception network can be the perception network in the above-mentioned embodiment.

[0316] like Fig.16 As shown in , the perceptual network training method includes:

[0317] 1601. Obtain a pre-labeled detection box of a target object in an image.

[0318] 1602. Obtain a target detection frame corresponding to the image and the first perception network, where the target detection frame is used to identify the target object.

[0319] In an embodiment of the present application, a detection result of the image can be obtained, where the detection result is obtained by performing object detection on the image through a first perception network, and the detection result includes a target detection box corresponding to the first object.

[0320] 1603. Iteratively train the first perception network according to a loss function to output a second perception network; wherein the loss function is related to an intersection over union (IoU) ratio between the pre-labeled detection box and the target detection box.

[0321] In an embodiment of the present application, the preset loss function is also related to a shape difference between the target detection box and the pre-labeled detection box, wherein the shape difference is negatively correlated with an area of ​​the pre-labeled detection box.

[0322] In an embodiment of the present application, the rectangular detection frame includes a first side and a second side connected to each other, the circumscribed rectangular frame includes a third side corresponding to the first side, and a fourth side corresponding to the second side, and the area difference is also positively correlated with the length difference between the first side and the third side, and positively correlated with the length difference between the second side and the fourth side.

[0323] In an embodiment of the present application, the preset loss function is also related to the position difference between the target detection box and the pre-labeled detection box in the image, wherein the position difference is negatively correlated with the area of ​​the pre-labeled detection box; or the position difference is negatively correlated with the area of ​​the minimum circumscribed rectangle of the convex hull of the pre-labeled detection box and the target detection box.

[0324] In an embodiment of the present application, the target detection frame includes a first corner point and a first center point, the pre-labeled detection frame includes a second corner point and a second center point, the first corner point and the second corner point are two endpoints of a rectangular diagonal, and the position difference is also positively correlated with the position difference between the first center point and the second center point in the image, and negatively correlated with the length of the first corner point and the second corner point.

[0325] Exemplarily, the loss function may be as follows:

[0326]

[0327]

[0328]

[0329] In the embodiment of the present application, the newly designed border regression loss function utilizes the IoU loss term with scale invariance, which is applied to the target detection measurement method, the loss term that considers the aspect ratio of the predicted border and the true border, and the loss term of the ratio of the distance between the center coordinate of the predicted border and the center coordinate of the true border and the distance between the lower right corner coordinate of the predicted border and the upper left corner coordinate of the true border. The IoU loss term naturally introduces a scale-invariant border prediction quality evaluation index. The loss term of the aspect ratio of the two borders measures the degree of fit of the shape between the two borders. The third distance ratio measurement term is used to solve the problem that when IoU=0, the relative position relationship between the predicted border and the true border cannot be known and it is difficult to perform back propagation and minimize the loss function. After the distance ratio is introduced, in order to make the distance ratio smaller, it will naturally pull o closer. p With o g The distance between and The three loss functions are assigned different weights to balance the impact of each item, among which the aspect ratio and distance ratio will introduce The border regression loss function proposed in this patent is applicable to various two-stage and one-stage algorithms, has good versatility, and plays an excellent role in promoting the object scale, border fit, and fit between the center point and the border corner point.

[0330] In the embodiment of the present application, the IoU, which measures the degree of fit between the detected border and the real border in target detection, is used as the loss function for border regression. Due to the scale invariance of IoU, the problem that other border regression functions are sensitive to scale changes in the past is solved; and the loss term of the aspect ratio between the predicted border and the real border is introduced to make the predicted border fit the real border more closely during the training process, and the loss term of the aspect ratio between the predicted border and the real border is introduced. The weight coefficient is proportional to the real frame center coordinates to reduce the impact of scale changes; the ratio of the distance between the predicted frame center coordinates and the real frame center coordinates to the distance between the predicted frame bottom right corner coordinates and the real frame top left corner coordinates is introduced to solve the problem that when IoU=0, the relative position and distance between the two frames cannot be known, and the back propagation cannot be performed. After the distance ratio is introduced, in order to make the distance ratio smaller, it will naturally pull o closer. p With o g The distance between and The distance between them encourages the border to change in the right direction.

[0331] The preset loss function includes a target loss term related to the position difference, and the target loss term changes with the change of the position difference; wherein, when the position difference is greater than a preset value, the change rate of the target loss term is greater than a first preset change rate; and / or, when the position difference is less than a preset value, the change rate of the target loss term is less than a second preset change rate.

[0332] Optionally, the bounding box regression loss function can also be the following loss function:

[0333]

[0334]

[0335] The above-mentioned border regression loss function utilizes the IoU loss term, which is scale-invariant and widely used in detection metric methods, the loss term that considers the aspect ratio of the predicted border and the true border, and the pull loss term that narrows the distance between the predicted border and the true border. The IoU loss term naturally introduces a scale-invariant border prediction quality evaluation indicator. The loss term of the aspect ratio of the two borders measures the degree of fit between the shapes of the two borders. The third distance ratio metric term is used to solve the problem that when IoU = 0, the relative position relationship between the predicted border and the true border cannot be known, and it is difficult to perform back propagation and minimize the loss function. Therefore, the third term is introduced to narrow the distance between the predicted border and the true border. pull loss term, in which the ratio of the area area1 of the rectangle formed by the diagonal of the center of the predicted border and the center of the real border to the area area2 of the minimum convex hull rectangle of the predicted border and the real border is used as the distance_IoU term, and f(x)=-ln(1-x) is used as the loss function calculation term. It can be seen that the distance_IoU variation range is [0,1). From the curve of f(x)=-ln(1-x), it can be seen that it has a tendency to converge rapidly between [0,1). Therefore, setting x to x=distance_IoU to obtain the third loss term can quickly close the predicted border and the real border.

[0336] Reference Fig.17 , Fig.17 The process diagram of an object detection method provided in an embodiment of the present application is as follows: Fig.17 As shown in , the object detection method includes:

[0337] 1701. Receive a first input image;

[0338] 1702. Perform convolution processing on the first image to generate a plurality of first feature maps;

[0339] It should be noted that the “performing convolution processing on the input image” here should not be understood as merely performing convolution processing on the input image. In some implementations, convolution processing, pooling operations, etc. may be performed on the input image.

[0340] It should be noted that the “convolution processing is performed on the first image to generate multiple first feature maps” here should not be understood as merely performing convolution processing on the first image multiple times, and each convolution processing can generate a first feature map, that is, it should not be understood as each first feature map is obtained based on the convolution processing on the first image, but, from a holistic perspective, the first image is the source of multiple first feature maps; in one implementation, the first image can be convolved to obtain a first feature map, and then the generated first feature map can be convolved to obtain another first feature map, and so on, to obtain multiple first feature maps.

[0341] It should be noted that a series of convolution processes may be performed on the input image. Specifically, in each convolution process, the first feature map obtained by the previous convolution process may be convolved to obtain a first feature map. In the above manner, multiple first feature maps may be obtained.

[0342] It should be noted that the multiple first feature maps may be feature maps with multi-scale resolutions, that is, the multiple first feature maps are not feature maps with the same resolution. In an optional implementation, the multiple first feature maps may constitute a feature pyramid.

[0343] Among them, an input image can be received, and a convolution process can be performed on the input image to generate multiple first feature maps with multi-scale resolutions; the convolution processing unit can perform a series of convolution processes on the input image to obtain feature maps at different scales (with different resolutions). The convolution processing unit can take many forms, such as visual geometry group (VGG), residual neural network (resnet), GoogLeNet's core structure (Inception-net), etc.

[0344] 1703. Generate a plurality of second feature maps according to the plurality of first feature maps; wherein the plurality of first feature maps include more texture details of the input image and / or position details in the input image than the plurality of second feature maps;

[0345] It should be noted that the “generating multiple second feature maps based on the multiple first feature maps” here should not be understood as the source of each second feature map in the multiple second feature maps is the multiple first feature maps; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on one or more first feature maps in the multiple first feature maps; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on one or more first feature maps in the multiple first feature maps, and other second feature maps except themselves; in one implementation, a part of the second feature maps in the multiple second feature maps are generated directly based on other second feature maps except themselves. At this time, since “other second feature maps except themselves are generated based on one or more first feature maps in the multiple first feature maps”, it can be understood as generating multiple second feature maps based on the multiple first feature maps.

[0346] It should be noted that the multiple second feature maps may be feature maps with multi-scale resolutions, that is, the multiple second feature maps are not feature maps with the same resolution. In an optional implementation, the multiple second feature maps may constitute a feature pyramid.

[0347] Among them, a convolution operation can be performed on the top-level feature map C4 among the multiple first feature maps generated by the convolution processing unit. Exemplarily, the number of channels of the top-level feature map C4 can be reduced to 256 by using void convolution and 1×1 convolution, as the top-level feature map P4 of the feature pyramid; the output result of the next layer feature map C3 of the top layer is horizontally linked and the number of channels is reduced to 256 by using 1×1 convolution, and then the output result is added to the feature map P4 channel by channel and pixel by pixel to obtain the feature map P3; and so on, from top to bottom, a first feature pyramid is constructed, and the first feature pyramid may include multiple second feature maps.

[0348] The texture details can be used to express the details of small targets and edge features. Compared with the second feature map, the first feature map includes more texture details, so that the detection accuracy of the detection results for small target detection is higher. The position details can be information expressing the position of objects in the image and the relative positions between objects.

[0349] Compared with the first feature map, multiple second feature maps can include more deep features. The deep features contain rich semantic information, which has a good effect on classification tasks. At the same time, the deep features have a large receptive field and can have a better detection effect on large targets. In one implementation, by introducing a top-down pathway to generate multiple second feature maps, the rich semantic information contained in the deep features can be naturally propagated downward, so that the second feature maps of each scale contain rich semantic information.

[0350] 1704. Generate a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps;

[0351] 1705. Output a first detection result of the object included in the image based on at least one third feature map among the multiple third feature maps.

[0352] In an existing implementation, the second feature map generation unit (such as a feature pyramid network) propagates the rich semantic information contained in the deep features downward by introducing a top-down path, so that the second feature maps of each scale contain rich semantic information. At the same time, the deep features have a large receptive field, so that a better detection effect can be achieved for large targets. However, in the existing implementation, the more refined position detail information and texture detail information contained in the shallower feature maps are ignored, which has a great impact on the detection accuracy of medium and small targets. In an embodiment of the present application, the second feature map generation unit introduces the shallow texture detail information of the original feature map (multiple first feature maps generated by the convolution processing unit) into the deep feature map (multiple second feature maps generated by the first feature map generation unit) to generate multiple third feature maps, and the third feature map with shallow rich texture detail information is used as the input data for the detection unit to perform target detection, which can improve the detection accuracy of subsequent object detection.

[0353] It should be noted that this embodiment does not mean that the detection accuracy of objects in any image including small targets will be higher, but for a large number of samples, this embodiment can have a higher comprehensive detection accuracy.

[0354] Optionally, the plurality of second feature maps include more semantic information than the plurality of first feature maps.

[0355] Optionally, the multiple first feature maps, the multiple second feature maps, and the multiple third feature maps are feature maps with multi-scale resolutions.

[0356] Optionally, the plurality of first feature maps include a first target feature map, the plurality of second feature maps include a second target feature map, the plurality of third feature maps include a third target feature map and a fourth target feature map, and the resolution of the third target feature map is smaller than that of the fourth target feature map; generating a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps includes:

[0357] Downsampling the third target feature map to obtain a fifth target feature map, wherein the fifth target feature map has the same number of channels and resolution as the second target feature map; downsampling and convolution processing are performed on the first target feature map to obtain a sixth target feature map, wherein the sixth target feature map has the same number of channels and resolution as the second target feature map; channel superposition of the fifth target feature map, the second target feature map and the sixth target feature map is performed to generate the fourth target feature map; or,

[0358] The third target feature map is downsampled to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; the first target feature map is downsampled to obtain a sixth target feature map, and the sixth target feature map has the same resolution as the second target feature map; the fifth target feature map, the second target feature map and the sixth target feature map are channel-superimposed and convolved to generate the fourth target feature map, and the fourth target feature map has the same number of channels as the second target feature map.

[0359] Optionally, the method further comprises:

[0360] Convolving at least one third feature map among the plurality of third feature maps through the first convolutional layer to obtain at least one fourth feature map;

[0361] and performing convolution on at least one third feature map among the plurality of third feature maps to obtain at least one fifth feature map; wherein the receptive field corresponding to the fourth feature map is larger than the receptive field corresponding to the fifth feature map;

[0362] Correspondingly, outputting a first detection result of the object included in the image according to at least one third feature map among the plurality of third feature maps includes:

[0363] A first detection result of the object included in the image is outputted according to the at least one fourth feature map and the at least one fifth feature map.

[0364] Optionally, the method further comprises:

[0365] Processing the fourth feature map according to the first weight value to obtain a processed fourth feature map;

[0366] Processing the fifth feature map according to the second weight value to obtain a processed fifth feature map; wherein, when the object to be detected included in the fourth feature map is larger than the object to be detected included in the fifth feature map, the first weight value is greater than the second weight value;

[0367] Correspondingly, outputting a first detection result of the object included in the image according to at least one third feature map among the plurality of third feature maps includes:

[0368] Outputting a first detection result of the object included in the image according to the processed fourth feature map and the processed fifth feature map.

[0369] Optionally, the first detection result includes a first detection frame, and the method further includes:

[0370] Acquire a second detection result of the first image, where the second detection result is obtained by performing object detection on the first image through a second perception network, the object detection accuracy of the first perception network is higher than the object detection accuracy of the second perception network, and the second detection result includes a second detection box, and there is an intersection between an area where the second detection box is located and an area where the first detection box is located;

[0371] If the ratio of the area of ​​the intersection to the area of ​​the first detection frame is smaller than a preset value, the second detection result is updated so that the updated second detection result includes the first detection frame.

[0372] Optionally, the second detection result includes multiple detection frames, and there is an intersection between the area where each detection frame in the multiple detection frames is located and the area where the first detection frame is located, and the multiple detection frames include the second detection frame, wherein among the areas of the intersections of the areas where each detection frame in the multiple detection frames is located and the areas where the first detection frame is located, the area of ​​the intersection between the areas where the second detection frame is located and the areas where the first detection frame is located is the smallest.

[0373] Optionally, the first image is an image frame in a video, the second image is an image frame in the video, and a frame spacing between the first image and the second image in the video is less than a preset value, and the method further includes:

[0374] Acquire a third detection result of the second image, where the third detection result includes a fourth detection frame and an object category corresponding to the fourth detection frame;

[0375] When a shape difference and a position difference between the fourth detection frame and the first detection frame are within a preset range, the first detection frame corresponds to the object category corresponding to the fourth detection frame.

[0376] Optionally, the detection confidence of the fourth detection frame is greater than a preset threshold.

[0377] Reference Fig.18 , Fig.18 A schematic diagram of a perceptual network training device provided in an embodiment of the present application is shown in FIG. Fig.18 As shown in FIG. 1 , the perceptual network training apparatus 1800 includes:

[0378] An acquisition module 1801 is used to acquire a pre-labeled detection frame of a target object in an image; and acquire a target detection frame corresponding to the image and the first perception network, wherein the target detection frame is used to identify the target object;

[0379] The iterative training module 1802 is used to iteratively train the first perception network according to a loss function to output a second perception network; wherein the loss function is related to the intersection over union (IoU) ratio between the pre-labeled detection box and the target detection box.

[0380] Optionally, the preset loss function is also related to a shape difference between the target detection box and the pre-labeled detection box, wherein the shape difference is negatively correlated with an area of ​​the pre-labeled detection box.

[0381] Optionally, the preset loss function is also related to the position difference between the target detection box and the pre-labeled detection box in the image, wherein the position difference is negatively correlated with the area of ​​the pre-labeled detection box; or the position difference is negatively correlated with the area of ​​the minimum circumscribed rectangle of the convex hull of the pre-labeled detection box and the target detection box.

[0382] Optionally, the target detection box includes a first corner point and a first center point, the pre-labeled detection box includes a second corner point and a second center point, the first corner point and the second corner point are two endpoints of a rectangular diagonal, and the position difference is also positively correlated with the position difference between the first center point and the second center point in the image, and negatively correlated with the length of the first corner point and the second corner point.

[0383] Optionally, the preset loss function includes a target loss term related to the position difference, and the target loss term changes as the position difference changes; wherein,

[0384] When the position difference is greater than a preset value, the change rate of the target loss item is greater than a first preset change rate; and / or, when the position difference is less than a preset value, the change rate of the target loss item is less than a second preset change rate.

[0385] Reference Fig.19 , Fig.19 An example of an object detection device provided in an embodiment of the present application is shown in FIG. Fig.19 As shown in FIG. 1 , the object detection device 1900 includes:

[0386] The receiving module 1901 is used to receive an input first image and perform convolution processing on the first image to generate a plurality of first feature maps;

[0387] A convolution processing module 1902, configured to perform convolution processing on the first image to generate a plurality of first feature maps;

[0388] A first feature map generating module 1903 is configured to generate a plurality of second feature maps according to the plurality of first feature maps; wherein the plurality of first feature maps include more texture details of the input image and / or position details in the input image than the plurality of second feature maps;

[0389] A second feature map generating module 1904, configured to generate a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps;

[0390] The detection module 1905 is used to output a first detection result of the object included in the image according to at least one third feature map among the multiple third feature maps.

[0391] Optionally, the plurality of second feature maps include more semantic information than the plurality of first feature maps.

[0392] Optionally, the multiple first feature maps, the multiple second feature maps, and the multiple third feature maps are feature maps with multi-scale resolutions.

[0393] Optionally, the plurality of first feature maps include a first target feature map, the plurality of second feature maps include a second target feature map, the plurality of third feature maps include a third target feature map and a fourth target feature map, the resolution of the third target feature map is smaller than that of the fourth target feature map, and the second feature map generating module is specifically configured to:

[0394] Downsampling the third target feature map to obtain a fifth target feature map, wherein the fifth target feature map has the same number of channels and resolution as the second target feature map; downsampling and convolution processing are performed on the first target feature map to obtain a sixth target feature map, wherein the sixth target feature map has the same number of channels and resolution as the second target feature map; channel superposition of the fifth target feature map, the second target feature map and the sixth target feature map is performed to generate the fourth target feature map; or,

[0395] The third target feature map is downsampled to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; the first target feature map is downsampled to obtain a sixth target feature map, and the sixth target feature map has the same resolution as the second target feature map; the fifth target feature map, the second target feature map and the sixth target feature map are channel-superimposed and convolved to generate the fourth target feature map, and the fourth target feature map has the same number of channels as the second target feature map.

[0396] Optionally, the device further comprises:

[0397] an intermediate feature extraction module, configured to convolve at least one of the plurality of third feature maps through a first convolutional layer to obtain at least one fourth feature map;

[0398] and performing convolution on at least one third feature map among the plurality of third feature maps to obtain at least one fifth feature map; wherein the receptive field corresponding to the fourth feature map is larger than the receptive field corresponding to the fifth feature map;

[0399] Accordingly, the detection module is specifically used for:

[0400] A first detection result of the object included in the image is outputted according to the at least one fourth feature map and the at least one fifth feature map.

[0401] Optionally, the device further comprises:

[0402] Processing the fourth feature map according to the first weight value to obtain a processed fourth feature map;

[0403] Processing the fifth feature map according to the second weight value to obtain a processed fifth feature map; wherein, when the object to be detected included in the fourth feature map is larger than the object to be detected included in the fifth feature map, the first weight value is greater than the second weight value;

[0404] Accordingly, the detection module is specifically used for:

[0405] Outputting a first detection result of the object included in the image according to the processed fourth feature map and the processed fifth feature map.

[0406] Optionally, the first detection result includes a first detection frame, and the acquisition module is further used to:

[0407] Acquire a second detection result of the first image, where the second detection result is obtained by performing object detection on the first image through a second perception network, the object detection accuracy of the first perception network is higher than the object detection accuracy of the second perception network, and the second detection result includes a second detection box, and there is an intersection between an area where the second detection box is located and an area where the first detection box is located;

[0408] An updating module is used to update the second detection result if the ratio of the area of ​​the intersection to the area of ​​the first detection frame is less than a preset value, so that the updated second detection result includes the first detection frame.

[0409] Optionally, the first image is an image frame in a video, the second image is an image frame in the video, a frame spacing between the first image and the second image in the video is less than a preset value, and the acquisition module is further used to:

[0410] Acquire a third detection result of the second image, where the third detection result includes a fourth detection frame and an object category corresponding to the fourth detection frame;

[0411] When a shape difference and a position difference between the fourth detection frame and the first detection frame are within a preset range, the first detection frame corresponds to the object category corresponding to the fourth detection frame.

[0412] Optionally, the detection confidence of the fourth detection frame is greater than a preset threshold.

[0413] Next, an execution device provided by an embodiment of the present application is introduced. Fig. 20 , Fig. 20 A schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 2000 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device, etc., which is not limited here. Among them, the execution device 2000 can be deployed with Fig.19 The object detection device described in the corresponding embodiment is used to implement Fig.19 The object detection function in the corresponding embodiment. Specifically, the execution device 2000 includes: a receiver 2001, a transmitter 2002, a processor 2003 and a memory 2004 (wherein the number of the processor 2003 in the execution device 2000 can be one or more, Fig. 20 In the example of FIG. 2003 , the processor 2003 may include an application processor 20031 and a communication processor 20032. In some embodiments of the present application, the receiver 2001, the transmitter 2002, the processor 2003 and the memory 2004 may be connected via a bus or other means.

[0414] The memory 2004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 2003. A portion of the memory 2004 may also include a non-volatile random access memory (NVRAM). The memory 2004 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0415] The processor 2003 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0416] The method disclosed in the above embodiment of the present application can be applied to the processor 2003, or implemented by the processor 2003. The processor 2003 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 2003. The above processor 2003 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 2003 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 2004, and the processor 2003 reads the information in the memory 2004 and completes the steps of the above method in combination with its hardware.

[0417] The receiver 2001 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 2002 can be used to output digital or character information through the first interface; the transmitter 2002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 2002 can also include a display device such as a display screen.

[0418] In one embodiment of the present application, the processor 2003 is used to execute Figures 9 to 11 The image processing method executed by the execution device in the corresponding embodiment. Specifically, the application processor 20031 is used to execute the object detection method in the above embodiment.

[0419] The present application also provides a training device. Fig.21 , Fig.21 is a structural diagram of a training device provided in an embodiment of the present application. The training device 2100 may be deployed with Fig.18 The perceptual network training device described in the corresponding embodiment is used to implement Fig.18 The functions of the perception network training device in the corresponding embodiment, specifically, the training device 2100 is implemented by one or more servers, and the training device 2100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 2121 (for example, one or more processors) and memory 2132, one or more storage media 2130 (for example, one or more mass storage devices) storing application programs 2142 or data 2144. Among them, the memory 2132 and the storage medium 2130 can be short-term storage or permanent storage. The program stored in the storage medium 2130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processor 2121 can be configured to communicate with the storage medium 2130 to execute a series of instruction operations in the storage medium 2130 on the training device 2100.

[0420] The training device 2100 may also include one or more power supplies 2126, one or more wired or wireless network interfaces 2150, one or more input and output interfaces 2158; or, one or more operating systems 2141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0421] In the embodiment of the present application, the central processor 2121 is used to execute the steps related to the perceptual network training method in the above embodiment.

[0422] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0423] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0424] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0425] For details, please refer to Fig. 22 , Fig. 22 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 2200, NPU 2200 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2203, which is controlled by the controller 2204 to extract matrix data from the memory and perform multiplication operations.

[0426] In some implementations, the operation circuit 2203 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2203 is a two-dimensional systolic array. The operation circuit 2203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2203 is a general-purpose matrix processor.

[0427] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2202 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2201 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 2208.

[0428] The unified memory 2206 is used to store input data and output data. The weight data is directly transferred to the weight memory 2202 through the direct memory access controller (DMAC) 2205. The input data is also transferred to the unified memory 2206 through the DMAC.

[0429] BIU stands for Bus Interface Unit, i.e., bus interface unit 2210 , which is used for interaction between AXI bus and DMAC and instruction fetch buffer (IFB) 2209 .

[0430] The bus interface unit 2210 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2209 to obtain instructions from the external memory, and is also used for the storage unit access controller 2205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0431] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2206 or to transfer weight data to the weight memory 2202 or to transfer input data to the input memory 2201.

[0432] The vector calculation unit 2207 includes multiple operation processing units, and further processes the output of the operation circuit 2203 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0433] In some implementations, the vector calculation unit 2207 can store the processed output vector to the unified memory 2206. For example, the vector calculation unit 2207 can apply a linear function; or a nonlinear function to the output of the operation circuit 2203, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2203, for example, for use in a subsequent layer in a neural network.

[0434] An instruction fetch buffer 2209 connected to the controller 2204 is used to store instructions used by the controller 2204;

[0435] Unified memory 2206, input memory 2201, weight memory 2202 and instruction fetch memory 2209 are all On-Chip memories. External memories are private to the NPU hardware architecture.

[0436] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0437] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0438] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0439] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0440] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data processing system, characterized in that: The data processing system comprises: a convolution processing unit, a first feature map generating unit, a second feature map generating unit and a detection unit, wherein the convolution processing unit is connected to the first feature map generating unit and the second feature map generating unit respectively, the first feature map generating unit is connected to the second feature map generating unit, and the second feature map generating unit is connected to the detection unit; The convolution processing unit is used to receive an input image and perform convolution processing on the input image to generate a plurality of first feature maps; the plurality of first feature maps include a first target feature map; The first feature map generating unit is used to generate a plurality of second feature maps according to the plurality of first feature maps; wherein the plurality of second feature maps include a second target feature map, the plurality of first feature maps include more texture details of the input image and / or position details in the input image than the plurality of second feature maps; the plurality of second feature maps include more semantic information and a larger receptive field than the plurality of first feature maps; The second feature map generating unit is used to generate a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps; the plurality of third feature maps include a third target feature map and a fourth target feature map, the resolution of the third target feature map is smaller than that of the fourth target feature map, and the second feature map generating unit is used to generate the fourth target feature map by the following steps: Downsampling the third target feature map to obtain a fifth target feature map, the fifth target feature map has the same number of channels and resolution as the second target feature map; downsampling and convolution processing are performed on the first target feature map to obtain a sixth target feature map, the sixth target feature map has the same number of channels and resolution as the second target feature map; channels are superimposed on the fifth target feature map, the second target feature map and the sixth target feature map to generate the fourth target feature map; or, downsampling the third target feature map to obtain a fifth target feature map, the fifth target feature map has the same number of channels and resolution as the second target feature map; downsampling the first target feature map to obtain a sixth target feature map, the sixth target feature map has the same resolution as the second target feature map; channels are superimposed on the fifth target feature map, the second target feature map and the sixth target feature map and convolution processing is performed to generate the fourth target feature map, the fourth target feature map has the same number of channels as the second target feature map; The detection unit is used to output a detection result of the object included in the image according to at least one third feature map among the multiple third feature maps.

2. The data processing system according to claim 1, characterized in that: The convolution processing unit is a backbone network, the first feature map generation unit and the second feature map generation unit are feature pyramid networks (FPNs), and the detection unit is a head end.

3. The data processing system according to claim 1 or 2, characterized in that: The data processing system further comprises: an intermediate feature extraction layer, configured to convolve at least one third feature map among the plurality of third feature maps to obtain at least one fourth feature map; and performing convolution on at least one third feature map among the plurality of third feature maps to obtain at least one fifth feature map; wherein the receptive field corresponding to the fourth feature map is larger than the receptive field corresponding to the fifth feature map; The detection unit is specifically configured to output a detection result of the object included in the image according to the at least one fourth feature map and the at least one fifth feature map.

4. The data processing system according to claim 3, characterized in that: The intermediate feature extraction layer is also used to: Processing the at least one fourth feature map according to the first weight value to obtain a processed fourth feature map; Processing the at least one fifth feature map according to the second weight value to obtain a processed fifth feature map; wherein, when the object to be detected included in the fourth feature map is larger than the object to be detected included in the fifth feature map, the first weight value is greater than the second weight value; Correspondingly, the detection unit is specifically configured to output a detection result of the object included in the image based on the processed fourth feature map and the processed fifth feature map.

5. An object detection method, characterized in that: The method is used in a first sensory network, and the method includes: Receiving an input first image, and performing convolution processing on the first image to generate a plurality of first feature maps; Generate multiple second feature maps according to the multiple first feature maps; wherein the multiple first feature maps include more texture details of the input image and / or position details in the input image than the multiple second feature maps; the multiple second feature maps include more semantic information and a larger receptive field than the multiple first feature maps; generating a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps; Outputting a first detection result of an object included in the image according to at least one third feature map among the plurality of third feature maps; the plurality of first feature maps include a first target feature map, the plurality of second feature maps include a second target feature map, the plurality of third feature maps include a third target feature map and a fourth target feature map, and the resolution of the third target feature map is smaller than that of the fourth target feature map; generating a plurality of third feature maps according to the plurality of first feature maps and the plurality of second feature maps comprises: Downsampling the third target feature map to obtain a fifth target feature map, wherein the fifth target feature map has the same number of channels and resolution as the second target feature map; downsampling and convolution processing are performed on the first target feature map to obtain a sixth target feature map, wherein the sixth target feature map has the same number of channels and resolution as the second target feature map; channel superposition of the fifth target feature map, the second target feature map and the sixth target feature map is performed to generate the fourth target feature map; or, The third target feature map is downsampled to obtain a fifth target feature map, and the fifth target feature map has the same number of channels and resolution as the second target feature map; the first target feature map is downsampled to obtain a sixth target feature map, and the sixth target feature map has the same resolution as the second target feature map; the fifth target feature map, the second target feature map and the sixth target feature map are channel-superimposed and convolved to generate the fourth target feature map, and the fourth target feature map has the same number of channels as the second target feature map.

6. The method according to claim 5, characterized in that The plurality of first feature maps, the plurality of second feature maps, and the plurality of third feature maps are feature maps with multi-scale resolutions.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: Convolving at least one third feature map among the plurality of third feature maps through the first convolutional layer to obtain at least one fourth feature map; and performing convolution on at least one third feature map among the plurality of third feature maps to obtain at least one fifth feature map; wherein the receptive field corresponding to the fourth feature map is larger than the receptive field corresponding to the fifth feature map; Correspondingly, outputting a first detection result of the object included in the image according to at least one third feature map among the plurality of third feature maps includes: A first detection result of the object included in the image is outputted according to the at least one fourth feature map and the at least one fifth feature map.

8. The method according to claim 7, characterized in that The method further comprises: Processing the fourth feature map according to the first weight value to obtain a processed fourth feature map; Processing the fifth feature map according to the second weight value to obtain a processed fifth feature map; wherein, when the object to be detected included in the fourth feature map is larger than the object to be detected included in the fifth feature map, the first weight value is greater than the second weight value; Correspondingly, outputting a first detection result of the object included in the image according to at least one third feature map among the plurality of third feature maps includes: Outputting a first detection result of the object included in the image according to the processed fourth feature map and the processed fifth feature map.

9. The method according to claim 5 or 6, characterized in that: The first detection result includes a first detection frame, and the method further includes: Acquire a second detection result of the first image, where the second detection result is obtained by performing object detection on the first image through a second perception network, the object detection accuracy of the first perception network is higher than the object detection accuracy of the second perception network, and the second detection result includes a second detection box, and there is an intersection between an area where the second detection box is located and an area where the first detection box is located; If the ratio of the area of ​​the intersection to the area of ​​the first detection frame is smaller than a preset value, the second detection result is updated so that the updated second detection result includes the first detection frame.

10. The method according to claim 9, characterized in that The second detection result includes multiple detection frames, and there is an intersection between the area where each detection frame in the multiple detection frames is located and the area where the first detection frame is located, and the multiple detection frames include the second detection frame, wherein among the areas of the intersections of the areas where each detection frame in the multiple detection frames is located and the area where the first detection frame is located, the area of ​​the intersection between the area where the second detection frame is located and the area where the first detection frame is located is the smallest.

11. The method according to claim 9, characterized in that The first image is an image frame in a video, the second image is an image frame in the video, a frame spacing between the first image and the second image in the video is less than a preset value, and the method further includes: Acquire a third detection result of the second image, where the third detection result includes a fourth detection frame and an object category corresponding to the fourth detection frame; When a shape difference and a position difference between the fourth detection frame and the first detection frame are within a preset range, the first detection frame corresponds to the object category corresponding to the fourth detection frame.

12. An object detection device, characterized in that: It comprises a storage medium, a processing circuit and a bus system; wherein the storage medium is used to store instructions, and the processing circuit is used to execute the instructions in the memory to perform the steps of the method described in any one of claims 5 to 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 5 to 11 are implemented.

Images