Object detection method and device, storage medium and electronic device

Abstract

The present application discloses an object detection method and device, a storage medium and an electronic device. The method includes: obtaining the initial image data to be detected; performing feature extraction and fusion operations on the initial image data to obtain the target image data; inputting the target image data into the target detection model to obtain the target detection result, wherein the target detection result includes the target detection frame, the target object category, and the target orientation category. The target detection model is used to predict the target image data, determine the target offset and the target rotation angle, the target offset is used to indicate the position of the target detection frame in the image coordinate system, and the target rotation angle is used to indicate the angle that the target detection frame needs to rotate in the image coordinate system. The present application solves the technical problem that the detection of the object orientation is not precise enough, resulting in the difficulty in ensuring the accuracy of the detection of the object orientation.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular, to an object detection method and device, a storage medium, and an electronic device. Background Art

[0002] At present, in the process of object detection, a single detection model is usually used to directly perform detection to output the detection result of the object. The detection result often focuses on the position of the object and cannot fully consider the attribute characteristics of the object. However, in application scenarios involving object detection, if the focus is on the position of the object and other attribute characteristics of the object are ignored, it will lead to inaccurate and incomplete detection results. In summary, there is a technical problem in the related technology that the detection of object orientation is not precise enough, which makes it difficult to guarantee the detection accuracy of object orientation.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an object detection method and device, a storage medium, and an electronic device to at least solve the technical problem that the detection of the object orientation is not precise enough, resulting in difficulty in ensuring the detection accuracy of the object orientation.

[0005] According to one aspect of an embodiment of the present application, there is provided an object detection method, comprising: acquiring initial image data to be detected, wherein the initial image data comprises a target object to be identified; performing feature extraction and fusion operations on the initial image data to obtain target image data, wherein a resolution of the target image data is smaller than a resolution of the initial image data; inputting the target image data into a target detection model to obtain a target detection result, wherein the target detection result comprises a target detection box, a target object category, and a target orientation category, the target detection model being used to predict the target image data, determine a target offset and a target rotation angle, the target detection box being used to indicate an image area in the target image data where the target object exists, the target object category being used to indicate an object category of the target object, the target orientation category being used to indicate a moving direction of the target object in an image coordinate system, the target offset being used to indicate a position of the target detection box in the image coordinate system, and the target rotation angle being used to indicate an angle by which the target detection box needs to be rotated in the image coordinate system.

[0006] According to another aspect of an embodiment of the present application, an object detection device is also provided, including: an acquisition module, used to acquire initial image data to be detected, wherein the initial image data includes a target object to be identified; a feature extraction and fusion module, used to perform feature extraction and fusion operations on the initial image data to obtain target image data, wherein the resolution of the target image data is smaller than the resolution of the initial image data; a detection module, used to input the target image data into a target detection model to obtain a target detection result, wherein the target detection result includes a target detection box, a target object category, and a target orientation category, the target detection model is used to predict the target image data, determine a target offset and a target rotation angle, the target detection box is used to indicate an image area in the target image data where the target object exists, the target object category is used to indicate an object category of the target object, the target orientation category is used to indicate a moving direction of the target object in an image coordinate system, the target offset is used to indicate a position of the target detection box in the image coordinate system, and the target rotation angle is used to indicate an angle by which the target detection box needs to be rotated in the image coordinate system.

[0007] Optionally, the device is used to input the target image data into a target detection model to obtain a target detection result in the following manner: obtaining a predetermined number of anchor frames, wherein the number of anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the target image data; detecting the target image data according to a predetermined total number of object categories and the number of anchor frames, and determining multiple prediction probabilities, wherein one prediction probability among the multiple prediction probabilities is used to indicate a prediction probability that an anchor frame corresponding to a sampling point includes the target object, and that the sampling point belongs to the object category corresponding to the anchor frame; determining the anchor frame corresponding to the prediction probability with the largest value among the multiple prediction probabilities as the target detection frame, and determining the object category corresponding to the target detection frame as the target object category.

[0008] Optionally, the device is used to input the target image data into a target detection model to obtain a target detection result in the following manner: detect the target image data to determine the coordinate information and size information of the minimum enclosing rectangle of the target object; generate a coded offset based on the coordinate information and the size information; perform a target decoding operation on the coded offset to obtain the target offset; determine the target rotation angle based on the minimum enclosing rectangle and the target detection frame, wherein the target rotation angle represents the rotation angle of the minimum enclosing rectangle relative to the target detection frame; and determine the category indicated by the angle between the orientation vector of the target object and the long side of the minimum enclosing rectangle as the target orientation category.

[0009] Optionally, the device is used to perform a sampling operation on the initial image data to obtain target image data in the following manner: perform a downsampling operation on the initial image data to determine a group of initial feature images, wherein the resolutions of each initial feature image in the group of initial feature images are different; perform a feature fusion operation on the group of initial feature images based on a feature pyramid to obtain a first group of fused feature images, wherein a fused feature image in the first group of fused feature images corresponds to an initial feature image in the group of initial feature images; perform the downsampling operation on the target initial feature image to obtain a second group of fused feature images, wherein the target initial feature image represents the image with the lowest resolution in the group of initial feature images; determine the target image data based on the first group of fused feature images and the second group of fused feature images.

[0010] Optionally, the device is also used for: inputting the target image data into the target detection model to obtain sample image data before obtaining the target detection result, wherein the sample image data is pre-labeled with position information of a sample minimum circumscribed rectangle associated with the sample object, a sample object category corresponding to the sample minimum circumscribed rectangle, a sample orientation category, and a sample rotation angle, wherein the sample rotation angle represents the rotation angle when the horizontal axis in the image coordinate system is rotated to be parallel to the wide side of the sample minimum circumscribed rectangle or the rotation angle when the vertical axis in the image coordinate system is rotated to be parallel to the long side of the sample minimum circumscribed rectangle; performing N rounds of training on the initial detection model according to the sample image data to obtain the target detection model, wherein the model parameters of the target detection model are the model parameters used in the i-th round of training of the initial detection model in the N rounds of training, and the i-th round of training represents the training round in which the model indicators of the initial detection model meet preset conditions, and each round of training determines the target loss value of the initial detection model, and adjusts the model parameters of the initial detection model based on the target loss value.

[0011] Optionally, the device is used to perform N rounds of training on an initial detection model according to the sample image data in the following manner to obtain the target detection model: perform N rounds of training on the initial detection model in the following manner, wherein each round of training includes the following steps: obtain a predetermined number of sample anchor frames, wherein the number of sample anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the sample image data; detect the target image data according to a predetermined total number of sample object categories and the number of sample anchor frames, and determine multiple sample prediction probabilities, wherein one sample prediction probability among the multiple sample prediction probabilities is used to indicate that a sample anchor frame corresponding to a sample sampling point includes the sample object, and the prediction probability that the sample sampling point belongs to the prediction object category corresponding to the sample anchor frame; determine a first loss value of the initial detection model based on the sample object category and the multiple sample prediction probabilities, and adjust the model parameters of the initial detection model based on the first loss value, wherein the target loss value includes the first loss value.

[0012] Optionally, the device is used to perform N rounds of training on an initial detection model according to the sample image data in the following manner to obtain the target detection model: perform N rounds of training on the initial detection model in the following manner, wherein each round of training includes the following steps: detect the sample image data to determine predicted coordinate information and predicted size information of a predicted minimum enclosing rectangle; generate a predicted offset for the predicted coordinate information and the predicted size information; determine a predicted rotation angle according to the sample minimum enclosing rectangle and the predicted minimum enclosing rectangle, wherein the predicted rotation angle represents the rotation angle of the sample minimum enclosing rectangle relative to the predicted minimum enclosing rectangle; determine a category indicated by an angle between the orientation vector of the sample object and the long side of the predicted minimum enclosing rectangle as a predicted orientation category; determine a second loss value according to the predicted offset, the predicted rotation angle, the sample offset and the sample rotation angle, determine a third loss value according to the predicted orientation category and the sample orientation category, and adjust model parameters of the initial detection model based on the second loss value and the third loss value, wherein the target loss value includes the second loss value and the third loss value.

[0013] Optionally, the device is used to determine the second loss value based on the predicted offset, the predicted rotation angle, the sample offset and the sample rotation angle in the following manner: performing a target encoding operation on the sample offset to obtain a sample encoding offset; determining the second loss value based on the sample encoding offset and the sample rotation angle together with the predicted offset and the predicted rotation angle; or performing a target decoding operation on the prediction offset to obtain a predicted decoding offset; determining the second loss value based on the sample offset and the sample rotation angle together with the predicted decoding offset and the predicted rotation angle.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned object detection method when running.

[0015] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above object detection method.

[0016] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the object detection method through the computer program.

[0017] In an embodiment of the present application, initial image data to be detected is obtained, wherein the initial image data includes a target object to be identified; feature extraction and fusion operations are performed on the initial image data to obtain target image data, wherein the resolution of the target image data is less than the resolution of the initial image data; the target image data is input into a target detection model to obtain a target detection result, wherein the target detection result includes a target detection frame, a target object category, and a target orientation category. The target detection model is used to predict the target image data, determine a target offset and a target rotation angle, the target detection frame is used to indicate an image area where a target object exists in the target image data, the target object category is used to indicate an object category of the target object, the target orientation category is used to indicate a moving direction of the target object in an image coordinate system, the target offset is used to indicate a position of the target detection frame in the image coordinate system, and the target rotation angle is used to indicate an angle by which the target detection frame needs to be rotated in the image coordinate system. That is, according to the target detection result of the above target detection model, the position, category and orientation information of the target object in the initial image data can be obtained, thereby achieving a technical effect of improving the accuracy and robustness of object detection, and further, solving the technical problem that the detection of the object orientation is not fine enough, resulting in difficulty in ensuring the detection accuracy of the object orientation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application environment of an optional object detection method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of an optional object detection method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional object detection method according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of another optional object detection method according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of another optional object detection method according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of another optional object detection method according to an embodiment of the present application;

[0025] Figure 7is a schematic diagram of another optional object detection method according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of another optional object detection method according to an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of another optional object detection method according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of another optional object detection method according to an embodiment of the present application;

[0029] Fig.11 is a schematic structural diagram of an optional object detection device according to an embodiment of the present application;

[0030] Fig.12 is a schematic structural diagram of an optional object detection product according to an embodiment of the present application;

[0031] Fig.13 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] The present application is described below in conjunction with embodiments:

[0035] According to one aspect of an embodiment of the present application, a method for object detection is provided. Optionally, in this embodiment, the object detection method can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1 As shown, the server 101 is connected to the terminal device 103 via a network, and can be used to provide services for the terminal device or the application installed on the terminal device. The application 107 can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The terminal device 103 may be a terminal configured with an application, and may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (Virtual Reality, VR for short) terminal, an augmented reality (Augmented Reality, AR for short) terminal, a mixed reality (Mixed Reality, MR for short) terminal and other computer devices. The above-mentioned server may be a single server, or a server cluster consisting of multiple servers, or a cloud server.

[0036] Combination Figure 1 As shown, the object detection method can be executed by an electronic device, which can be a terminal device or a server. The object detection method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0037] The above is only an example and is not specifically limited in this embodiment.

[0038] Optionally, as an optional implementation, as Figure 2 As shown, the above object detection method includes:

[0039] S202, obtaining initial image data to be detected, wherein the initial image data includes a target object to be identified;

[0040] Optionally, in an embodiment of the present application, the above-mentioned initial image data may include but is not limited to portrait images, pet images, device images, landscape images, etc., and the initial image data can be obtained in the following ways, including but not limited to using an image acquisition device to acquire the initial image data in real time and uploading it to an image storage device, or, using an image acquisition device to acquire a video, obtaining the initial image data by intercepting video frames, and uploading the initial image data to an image storage device, or, using an image acquisition device to acquire a video, uploading the video to an image storage device, and having the image storage device intercept video frames to obtain the initial image data, wherein the image acquisition device may include but is not limited to a smart phone, a computer, a notebook, a smart wearable device, a camera, etc., and the image storage device may include but is not limited to a server, a computer, a laptop, a smart phone, a tablet computer, a digital camera, a scanner, a printer, etc.

[0041] Optionally, in an embodiment of the present application, the target object may include but is not limited to portraits, pets, equipment, vehicles, buildings, etc.

[0042] Exemplarily, a camera is set at a traffic intersection, and the initial image data is acquired through the camera. The target objects may include but are not limited to vehicles, pedestrians, traffic signs, etc.

[0043] It should be noted that the initial image data may include image data of a training or prediction reasoning part, and the initial image may not contain a target object to be identified, and the initial image data is a background image.

[0044] S204, performing a sampling operation on the initial image data to obtain target image data, wherein the resolution of the target image data is smaller than the resolution of the initial image data;

[0045] Optionally, in an embodiment of the present application, the target image data may be understood as image data obtained by reducing the resolution of the initial image data, that is, the target image data has a lower resolution and a faster processing speed than the initial image data.

[0046] In an exemplary embodiment, the resnet50 network module and the feature pyramid network module are used to perform feature extraction and fusion operations on the initial image data to obtain image data of different resolutions. For example, the initial image data is a picture with a resolution of 1000. 1000 images A, specifically:

[0047] S1, input image A into the resnet50 network module;

[0048] S2, obtain the feature images C3, C4, and C5 output by the feature extraction links of image A at different stages in the resnet50 network module, wherein the feature image C3 is downsampled 3 times relative to image A, the feature image C4 is downsampled 4 times relative to image A, and the feature image C5 is downsampled 5 times relative to image A;

[0049] For example, after N times of residual network extraction on the initial image data, a convolution with a stride of 2 can be used for downsampling.

[0050] S3, input feature image C3, feature image C4, and feature image C5 into the feature pyramid network module, and perform the following operations respectively:

[0051] S3-1, performing a convolution operation on C5 to obtain a fused feature image P5, and performing two downsampling operations on the feature image C5 to obtain feature images P6 and P7;

[0052] S3-2, perform a convolution operation on C5 to obtain C5'' with the same number of channels as C4, add C5'' and C4 to obtain P4', and then perform a convolution operation on P4' to obtain a fused feature image P4;

[0053] S3-3, upsample C4 once to obtain a feature image C4' with the same resolution as C3, perform a convolution operation on the feature image C4' to obtain C4'' with the same number of channels as C3, add C4'' and C3 to obtain P3', and then perform a convolution operation on P3' to obtain a fused feature image P3; wherein the convolution kernel size of the above convolution can be flexibly set, and this application does not impose any limitation on this.

[0054] S4, determining the fused characteristic image P3, the fused characteristic image P4, the fused characteristic image P5, the characteristic image P6 and the characteristic image P7 as the above-mentioned target image data.

[0055] S206, input the target image data into the target detection model to obtain the target detection result, wherein the target detection result includes a target detection frame, a target object category, and a target orientation category. The target detection model is used to predict the target image data and determine a target offset and a target rotation angle. The target detection frame is used to indicate an image area where a target object exists in the target image data. The target object category is used to indicate an object category of the target object. The target orientation category is used to indicate a moving direction of the target object in an image coordinate system. The target offset is used to indicate a position of the target detection frame in the image coordinate system. The target rotation angle is used to indicate an angle by which the target detection frame needs to be rotated in the image coordinate system.

[0056] Optionally, in an embodiment of the present application, the target detection model may include but is not limited to a neural network model architecture. The position, category and orientation information of the target object may be obtained through the target detection results. The target detection box may be used to indicate the image area where the target object is located. The target object category indicates the category of the detected target object, and the target orientation category indicates the moving direction of the target object in the image coordinate system.

[0057] For example, Figure 3 is a schematic diagram of an optional object detection method according to an embodiment of the present application, such as Figure 3 As shown, target object A and target object B are included. Target object A corresponds to target detection frame A, target object category is car category, and target orientation category is moving right, which is the direction indicated by arrow line A; target object B corresponds to target detection frame B, target object category is truck category, and target orientation category is moving left, which is the direction indicated by arrow line B.

[0058] Optionally, in an embodiment of the present application, the above-mentioned target offset refers to the position offset of the target detection frame relative to the image boundary or other reference object, and the target rotation angle refers to the rotation angle of the target detection frame relative to the x-axis or y-axis of the image coordinate system. The origin of the above-mentioned image coordinate system may include but is not limited to being located in the upper left corner of the image, with the x-axis extending to the right and the y-axis extending downward. It should be noted that the target detection frame here is a detection frame after the size, position, direction and other information have been adjusted.

[0059] For example, Figure 4 is a schematic diagram of another optional object detection method according to an embodiment of the present application, such as Figure 4 As shown, the image coordinate system is shown as Figure 4 As shown, including target object 402, the value range of the above target rotation angle may include but is not limited to [-π / 2,π / 2), wherein, if the x-axis rotates counterclockwise, the value of the target rotation angle is negative, and if it rotates clockwise, the value of the target rotation angle is positive. The target object 402 is obtained by rotating counterclockwise relative to the x-axis, and when the x-axis is rotated counterclockwise to a direction parallel to the wide side of the target detection frame, the rotation angle is theta1, and the direction is to the upper right. Theta1 is the above target rotation angle, and the wide side of the target detection frame is longer than the high side of the target detection frame. It should be noted that the target detection frame here is a detection frame after the size, position, direction and other information have been adjusted.

[0060] For example, Figure 5 is a schematic diagram of another optional object detection method according to an embodiment of the present application, such as Figure 5 As shown, the image coordinate system is shown as Figure 5As shown, the target object 502 is obtained by rotating the target object 502 clockwise relative to the x-axis, and when the x-axis is rotated clockwise to a direction parallel to the wide side of the target detection frame, the rotation angle is theta2, and the direction is downward and right, theta2 is the above-mentioned target rotation angle, and the wide side of the target detection frame is longer than the high side of the target detection frame. It should be noted that the target detection frame here is a detection frame after the size, position, direction and other information have been adjusted.

[0061] In an exemplary embodiment, Figure 6 is a schematic diagram of another optional object detection method according to an embodiment of the present application, such as Figure 6 As shown in FIG. 1 , taking one of the sampling points A as an example, the sampling point A corresponds to multiple anchor frames. The target offset can be understood as follows: during the detection process of the target detection model, the anchor with the highest degree of overlap with the target object is determined from a number of preset anchor frames, and it is determined as the target detection frame (the target detection frame here has not been adjusted in size, position, direction, etc.). Then, the target offset predicted by the target detection model is used to adjust the position, size, direction, etc. of the target detection frame. For example, the target offset is expressed as (t x , t y , t w , t h ), represents the change in the position and size of the minimum bounding rectangle relative to the target detection box. Specifically, t x and t y Indicates the horizontal and vertical offset of the center point of the rotated rectangle relative to the center point of the anchor, t w and t h Indicates the scaling ratio of the width and height of the rotated rectangle relative to the anchor. That is, the minimum bounding rectangle refers to the detection frame after the target detection frame is adjusted according to the target offset and the target rotation angle.

[0062] It should be noted that the method may include but is not limited to calculating the IoU (Intersection over Union) between the target object and each anchor to measure the degree of overlap between the anchor box and the ground truth box.

[0063] In an exemplary embodiment, Figure 7 is a schematic diagram of another optional object detection method according to an embodiment of the present application. The object detection method proposed in the present application can be applied to image processing, target detection, rotation target detection, orientation detection, target orientation detection based on deep learning, etc., and is illustrated by combining with traffic application scenarios. Figure 7 As shown, the following steps may be included but not limited to:

[0064] S702, start;

[0065] S704, acquiring collected traffic image data (the above-mentioned initial image data), the traffic image data including but not limited to vehicle driving images, traffic congestion images, traffic light images, traffic road condition images, etc.;

[0066] S706, processing the traffic image data, including but not limited to image denoising, image grayscale, image normalization, etc.

[0067] S708, inputting the traffic image data into the target detection model, performing the above sampling operation in the target detection model to obtain the above target image data, and then performing reasoning on the target image data;

[0068] S710, obtaining a target detection frame, a target object category, and a coded target offset and a target angle, decoding the coded target offset, adjusting a size of the target detection frame according to the decoded target offset, adjusting a position of the target detection frame according to the target rotation angle, etc., to obtain a minimum bounding rectangle;

[0069] Among them, the encoded target offset is expressed as ( ) as an example, the decoding operation is performed according to the following formula:

[0070]

[0071] in, are the central horizontal coordinate, vertical coordinate, width and height of the target detection frame. are the central abscissa, ordinate, width, and height of the above minimum circumscribed rectangle, where: express The exponential power of e, express The exponential power of e.

[0072] S712, end.

[0073] In an exemplary embodiment, it is assumed that the original resolution of the initial image corresponding to the initial image data is 1080 1920 3 (1080 is the number of pixels in the height of the initial image, 1920 is the number of pixels in the width of the initial image, and 3 is the number of channels (the initial image here includes three channels: R, red, G, green, and B). Then, the initial image is scaled to 640×640. During the scaling process, the aspect ratio of the initial image remains unchanged. Therefore, 0 values ​​can be filled in the height direction of the initial image to make the initial image 640×640. Then, the target recognition network extracts features from the 640×640 target image data to obtain feature images of different resolutions (such as 80×80, 40×40, and 20×20).

[0074] Furthermore, each feature subgrid (the above sampling point) in the 80×80 feature map image is used as an anchor point. On this anchor point, anchor frames with different aspect ratios and sizes can be preset manually. Assume that the base size is 32, and the range of anchor frame size is: 32×2 0 , 32×2 1 / 3 , 32×2 2 / 3 , that is, the anchor box base size is 32×2 0 , 32 pixels; 32×2 1 / 3 2 representing the base size 1 / 3 ; 32×2 2 / 3 2 representing the base size 2 / 3 ; If the aspect ratio is 1:2, 2:1, 1:1, then we can get 9 anchor boxes of different sizes, which are:

[0075] The base size is 32 pixels, and the aspect ratios of the three anchor boxes are 1:2, 2:1, and 1:1;

[0076] Three anchor boxes with a size of 32×7.0 pixels and aspect ratios of 1:2, 2:1, and 1:1;

[0077] Three anchor boxes with a size of 32×7.333 pixels and aspect ratios of 1:2, 2:1, and 1:1;

[0078] At this time, for the 80×80 feature image, each sampling segment has these 9 anchor boxes. For example:

[0079] For a 40×40 feature image, the base size of its anchor frame can be different. For example, 64 is used as the base size of the anchor frame. In this case, the base size is 64 pixels, so that a larger feature image can detect a smaller target, and a smaller feature image can detect a larger target. For example, the content of one sampling point on a 80×80 feature image is the same as the content of 8 sampling points on a 640×640 feature image.

[0080] Furthermore, the N×X3×80×80 feature image in the target image data obtained by downsampling an initial image corresponding to the initial image data by 8 times is used as the input of the detection head in the target recognition model, and the results corresponding to the three branches are obtained, corresponding to: regression branch N×(9×5)×80×80, category classification branch N×(cls_num×9)×80×80 and orientation classification branch N×(9×4)×80×80, where the 4×9 represents the four different orientations of the anchor box prediction, and the "5" in the regression branch represents the offsets that need to be predicted for each anchor box:

[0081] Δx: offset of the center point of the anchor box on the x-axis; Δy: offset of the center point of the anchor box on the y-axis; Δw: ratio of the width of the anchor box to the base width of the anchor box; Δh: ratio of the height of the anchor box to the base height of the anchor box; Δθ: offset of the rotation angle of the anchor box (if the anchor box is rotated), cls_num is the number of categories predicted by the target recognition model.

[0082] Specifically, for each anchor point in the feature image (the sampling point mentioned above), the classification branch in the target recognition network predicts the probability that each of the 9 anchor boxes contains the target category. For example, if cls_num is 3 (representing the three categories of people, vehicles, and animals), then for each anchor point, the network predicts the probability that each of the 9 anchor boxes belongs to these 3 categories, that is, a total of 27 probability values ​​are predicted (9 anchor boxes × 3 categories).

[0083] Furthermore, non-maximum suppression (NMS) can be used to perform unified global processing on the decoded prediction box to obtain the prediction result. Specifically: the target recognition network pre-sets a target threshold in the inference stage. Taking a sampling point as an example, the sampling point has 9 anchor boxes. For each anchor box, 5 values ​​of the offset, 3 values ​​of the predicted category and four probability values ​​of the orientation are predicted. Taking the predicted value of each anchor box as an example, the 5 offset values ​​are decoded relative to the corresponding anchor box to obtain a rectangle with a rotation angle (that is, a prediction box). The prediction box has three categories of prediction probabilities. If all three prediction probabilities are less than the target threshold, the prediction box will be determined as the background. If one of the three prediction probabilities is greater than the target threshold, the prediction box will be determined as the category corresponding to the prediction probability greater than the target threshold. The prediction box is classified according to the predicted category so that each category can be independently NMSed to determine the above target category. Taking one of the sampling points as an example, the specific steps are as follows:

[0084] S1, set the category prediction threshold and intersection over union (IoU) threshold of NMS input;

[0085] S2, remove the prediction boxes that are lower than the category prediction threshold in each prediction category corresponding to the 9 prediction boxes corresponding to the sampling point;

[0086] Next, take one of the prediction categories as an example and perform the following operations:

[0087] S3, select the highest probability box: select the box with the highest prediction probability from the remaining prediction boxes corresponding to the prediction category as the reference box of the prediction category;

[0088] S4, calculate the intersection over union (IoU) between the reference box and other predicted boxes of the same category;

[0089] S5, generating the IoU between each prediction box and the reference box in other prediction boxes of the same category in S4, and deleting the prediction box when the IoU between a prediction box and the reference box exceeds the preset intersection-over-union (IoU) threshold in S1;

[0090] S6, repeat the above steps S3 to S5 until there are no more predicted boxes or a new reference box cannot be selected;

[0091] S7, after NMS processing, each prediction category will retain the prediction box with higher prediction probability and lower overlap as the above target category.

[0092] In an exemplary embodiment, there are four prediction boxes on the feature image, each of which predicts the prediction probabilities of three categories, as shown in the following table:

[0093]

[0094] First, assuming that the target threshold is 0.5, the category A score of prediction box 1 is the highest. Therefore, prediction box 1 is processed and prediction box 1 is selected as the candidate box. Prediction box 1 and prediction box 4 predict the same category A, and their IoU (assuming it is 0.8) is greater than the target threshold. Therefore, the prediction of prediction box 4 is deleted. Next, prediction box 2 is selected as the next candidate box because its category B score is the highest and no other prediction box predicts category B. Therefore, the corresponding category of prediction box 2 is retained as category B; and the category C score of prediction box 3 is the highest, and no other prediction box predicts category C. Therefore, the corresponding category of prediction box 3 is retained as category C.

[0095] Through the embodiments of the present application, initial image data to be detected is obtained, wherein the initial image data includes a target object to be identified; feature extraction and fusion operations are performed on the initial image data to obtain target image data, wherein the resolution of the target image data is less than the resolution of the initial image data; the target image data is input into a target detection model to obtain a target detection result, wherein the target detection result includes a target detection frame, a target object category, and a target orientation category. The target detection model is used to predict the target image data, determine a target offset and a target rotation angle, the target detection frame is used to indicate an image area where a target object exists in the target image data, the target object category is used to indicate an object category of the target object, the target orientation category is used to indicate a moving direction of the target object in an image coordinate system, the target offset is used to indicate a position of the target detection frame in the image coordinate system, and the target rotation angle is used to indicate an angle by which the target detection frame needs to be rotated in the image coordinate system. That is, according to the target detection result of the above target detection model, the position, category and orientation information of the target object in the initial image data can be obtained, thereby achieving a technical effect of improving the accuracy and robustness of object detection, and further, solving the technical problem that the detection of the object orientation is not fine enough, resulting in difficulty in ensuring the detection accuracy of the object orientation.

[0096] As an optional solution, the inputting the target image data into a target detection model to obtain a target detection result includes: obtaining a predetermined number of anchor frames, wherein the number of anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the target image data; detecting the target image data according to a predetermined total number of object categories and the number of anchor frames, and determining a plurality of prediction probabilities, wherein a prediction probability among the plurality of prediction probabilities is used to indicate that an anchor frame corresponding to a sampling point includes the target object, and the prediction probability that the sampling point belongs to the object category corresponding to the anchor frame; determining the anchor frame corresponding to the prediction probability with the largest value among the plurality of prediction probabilities as the target detection frame, and determining the object category corresponding to the target detection frame as the target object category.

[0097] Optionally, in an embodiment of the present application, the number of the above-mentioned anchor boxes can be flexibly set, and the present application does not impose any limitation on this. The above-mentioned sampling points can be understood as pixel points in the above-mentioned target image data, and the above-mentioned prediction probability refers to the prediction probability that the anchor box corresponding to the sampling point corresponds to the object category.

[0098] Exemplarily, for each anchor box corresponding to each pixel point, there are probability values ​​of the total number of object categories mentioned above, which respectively represent the predicted probability that the anchor box belongs to each target category. Assuming that the target image data has a total of 9 sampling points and the total number of object categories is 5, and each sampling point position corresponds to 3 anchor boxes, the category prediction probability corresponding to each anchor box corresponding to each sampling point can be calculated.

[0099] Specifically, assuming that the target image data is represented as a feature image of size 80×80, there are 6400 sampling points. For each sampling point, there are 9 preset anchors and 5 object categories. Then, taking one of the sampling points as an example, the category score corresponding to each anchor box corresponding to the sampling point can be represented by a 9×5 feature matrix, each row represents the prediction of an anchor box as the score of each category, and each column represents the score of each anchor box predicted as the category. The row of the corresponding category is found from the given 9×5 matrix, and the score of the corresponding anchor box in the row is taken out, including but not limited to using a softmax function to convert the matrix into the prediction probability corresponding to the anchor box, and then, the category corresponding to the prediction box whose prediction probability is greater than the above-mentioned target threshold is used as the above-mentioned target object category. It can be understood that a prediction box can include but is not limited to predicting multiple categories, and this application is not limited to this.

[0100] It should be noted that for each anchor box (that is, the row element in the above feature matrix), the category with the maximum score can be taken as the category detected by the anchor box (if the maximum value is less than the preset target threshold, the anchor box is predicted as a negative sample, that is, predicted as the background).

[0101] Through the embodiments of the present application, a predetermined number of anchor frames is obtained; the target image data is detected according to the predetermined total number of object categories and the number of anchor frames to determine multiple prediction probabilities; the anchor frame corresponding to the prediction probability with the largest value among the multiple prediction probabilities is determined as the target detection frame, and the object category corresponding to the target detection frame is determined as the target object category, and the position of the target detection frame and the corresponding object category are accurately determined, thereby achieving the technical effect of precision and accuracy in target detection.

[0102] As an optional scheme, the target image data is input into a target detection model to obtain a target detection result, including: detecting the target image data to determine the coordinate information and size information of the minimum enclosing rectangle of the target object; generating a coded offset according to the coordinate information and the size information; performing a target decoding operation on the coded offset to obtain the target offset; determining the target rotation angle according to the minimum enclosing rectangle and the target detection frame, wherein the target rotation angle represents the rotation angle of the minimum enclosing rectangle relative to the target detection frame; and determining the category indicated by the angle between the orientation vector of the target object and the long side of the minimum enclosing rectangle as the target orientation category.

[0103] Optionally, in the embodiment of the present application, the minimum bounding rectangle may be understood as a detection frame after the target detection frame is adjusted according to the target offset and the target rotation angle, which refers to a rectangular area enclosing the target object within the minimum rectangular frame. The target decoding operation may include but is not limited to being performed in the following manner:

[0104] The above encoding offset is expressed as ( ) as an example, the decoding operation is performed according to the following formula to obtain the above target offset, and then the minimum bounding rectangle is adjusted based on the target offset to obtain the target detection frame, including but not limited to:

[0105]

[0106] in, The horizontal coordinate, vertical coordinate, width and height of the center of the target detection frame;

[0107] are the central abscissa, ordinate, width, and height of the above minimum circumscribed rectangle, where: express The exponential power of e, express The exponential power of e.

[0108] Optionally, in an embodiment of the present application, the above-mentioned target rotation angle can be understood as the rotation angle between the above-mentioned minimum circumscribed rectangle and the above-mentioned target detection frame, that is, the target detection frame is rotated, and the required rotation angle is the above-mentioned target rotation angle. The area obtained after the rotation is the area corresponding to the above-mentioned minimum circumscribed rectangle. The above-mentioned target object's orientation vector can be understood as the direction facing the target object. The orientation vector can be used to describe the orientation, movement direction, action direction, etc. of the target object.

[0109] In an exemplary embodiment, assuming that a target detection frame includes a car (the target object mentioned above), and the orientation of the car is tilted, first, the rotation angle of the minimum circumscribed rectangle relative to the target detection frame is calculated, that is, the tilt angle of the car; then, the orientation category of the car is determined according to the angle between the orientation vector of the target object and the long side of the minimum circumscribed rectangle. Specifically, it may include but is not limited to: if the angle between the orientation vector of the car and the long side of the minimum circumscribed rectangle is a preset angle, then the orientation category of the car can be determined as "forward"; if the angle is another preset angle, then the orientation category of the car can be determined as "reversing"; it may also include but is not limited to the angle being within a preset angle range, and the orientation category of the car can be determined as "reversing" or "forward". This application does not impose any limitations on this.

[0110] For example, Figure 8 is a schematic diagram of another optional object detection method according to an embodiment of the present application, such as Figure 8 As shown, it is assumed that the rotation angle range of the X-axis is [-π / 2,π / 2), including but not limited to:

[0111] S802-a, the angle category of the orientation vector of the target object relative to the long side of the minimum circumscribed rectangle is 0 degrees, which can be mapped to the 0 orientation category, indicating that when the X-axis rotates clockwise to be parallel to the long side of the minimum circumscribed rectangle of the target object, after the rotation stops, the X-axis orientation is lower right, and the orientation of the target object is also lower right, that is, the orientation of the long side is the same as the orientation of the target object;

[0112] S802-b, the angle category of the orientation vector of the target object relative to the long side of the minimum circumscribed rectangle is 90 degrees, which can be mapped to orientation category 1, indicating that when the X-axis rotates clockwise to be parallel to the long side of the minimum circumscribed rectangle of the target object, after the rotation stops, the X-axis orientation is lower right, and the orientation of the target object is lower left, that is, the orientation angle between the long side orientation and the target object relative to the clockwise direction is 90 degrees;

[0113] S802-c, the angle category of the orientation vector of the target object relative to the long side of the minimum circumscribed rectangle is 180 degrees, which can be mapped to orientation category 1. At this time, assuming that the X-axis rotates clockwise to be parallel to the long side of the minimum circumscribed rectangle, it needs to rotate 270 degrees, which will cause the rotation range of [-π / 2,π / 2) to be exceeded. Therefore, the X-axis needs to rotate counterclockwise, which means that when the X-axis rotates counterclockwise to be parallel to the long side of the minimum circumscribed rectangle of the target object, after stopping the rotation, the X-axis is facing the upper right, and the orientation of the target object is facing the lower left. The orientation angle between the long side and the target object relative to the counterclockwise direction is 180 degrees;

[0114] S802-d, the angle category of the target object's orientation vector relative to the long side of the minimum circumscribed rectangle is 270 degrees, which can be mapped to orientation category 1. At this time, assuming that the X-axis rotates clockwise to be parallel to the long side of the minimum circumscribed rectangle, it needs to rotate 270 degrees, which will cause the rotation range of [-π / 2,π / 2) to exceed. Therefore, the X-axis will rotate counterclockwise to be parallel to the long side of the minimum circumscribed rectangle of the target object. After stopping the rotation, the X-axis is facing the upper right, while the target object is facing the lower right. The long side direction and the target object relative to the counterclockwise direction have an orientation angle of 270 degrees.

[0115] It should be noted that in the embodiment of the present application, the Y-axis can also be used for rotation to determine the above-mentioned target orientation category. The specific determination method is the same as the method of determining the above-mentioned target orientation category by rotating the X-axis. The present application does not make any specific limitation on the rotation of the coordinate axis.

[0116] In addition, the above rotation angle can also be 0, that is, the angle when the x-axis is tangent to or parallel to the wide side in the clockwise or counterclockwise direction is 0, 2 / π, such as Figure 8 S802 - e and S802 - f shown in the figure indicate that the minimum circumscribed rectangle is a regular rectangle.

[0117] In an exemplary embodiment, Fig. 9 is a schematic diagram of another optional object detection method according to an embodiment of the present application, such as Fig. 9 As shown, the target image data includes a target object A and a target object B. The target object A corresponds to a minimum circumscribed rectangle A, a direction A, and a width A of the minimum circumscribed rectangle. The width A of the minimum circumscribed rectangle of the target object A is obtained by rotating the X-axis counterclockwise. The direction after the rotation of the X-axis is the upper right, and the direction A is the lower left. The angle between the two is 180 degrees. Therefore, the angle category of the direction vector A of the target object relative to the long side of the minimum circumscribed rectangle A is 1, that is, the target direction category is the 1 direction category; and the target object B corresponds to the minimum circumscribed rectangle B, the direction B, and the width B of the minimum circumscribed rectangle. The width B of the minimum circumscribed rectangle of the target object B is obtained by rotating the X-axis counterclockwise. The direction after the rotation of the X-axis is the upper right, and the direction B is also the upper right. The angle between the two is 0 degrees. Therefore, the angle category of the direction vector B of the target object relative to the long side of the minimum circumscribed rectangle B is 0, that is, the target direction category is the 0 direction category.

[0118] Through the embodiments of the present application, the target image data is detected to determine the coordinate information and size information of the minimum bounding rectangle of the target object; a coded offset is generated based on the coordinate information and size information; a target decoding operation is performed on the coded offset to obtain a target offset; the target rotation angle is determined based on the minimum bounding rectangle and the target detection frame; the category indicated by the angle between the orientation vector of the target object and the long side of the minimum bounding rectangle is determined as the target orientation category, and the rotation angle and orientation category of the target are determined based on the target detection frame and the minimum bounding rectangle, thereby more accurately understanding and describing the position and orientation of the target in the image.

[0119] As an optional scheme, performing a sampling operation on the initial image data to obtain target image data includes: performing a downsampling operation on the initial image data to determine a group of initial feature images, wherein the resolutions of each initial feature image in the group of initial feature images are different; performing a feature fusion operation on the group of initial feature images based on a feature pyramid to obtain a first group of fused feature images, wherein a fused feature image in the first group of fused feature images corresponds to an initial feature image in the group of initial feature images; performing the downsampling operation on the target initial feature image to obtain a second group of fused feature images, wherein the target initial feature image represents the image with the lowest resolution in the group of initial feature images; and determining the target image data based on the first group of fused feature images and the second group of fused feature images.

[0120] Optionally, in an embodiment of the present application, different initial feature images in the above-mentioned set of initial feature images can be understood as images obtained after performing different times of downsampling on the initial image data, and the above-mentioned feature fusion operation refers to fusing the above-mentioned initial feature images of different resolutions to improve the performance and generalization ability of the target detection model, which can be achieved by including but not limited to feature combination, feature intersection, feature selection and the like.

[0121] Exemplarily, assuming that the initial image data is an image of 500×500 pixels, a downsampling operation is performed on the image to obtain a set of initial feature images. Specifically, the image can be downsampled into three images of 250×250 pixels, 125×125 pixels, and 50×50 pixels, which represent feature images at different resolutions, respectively. That is, the above set of initial feature images can be used to extract and analyze features at different scales.

[0122] In an exemplary embodiment, assuming that the above-mentioned set of initial feature images are feature images C3, feature images C4, and feature images C5, it may include but is not limited to inputting feature images C3, feature images C4, and feature images C5 into the feature pyramid network module, and performing the following operations respectively:

[0123] S1, perform a convolution operation on C5 to obtain a fused feature image P5, and perform two downsampling operations on the feature image C5 to obtain feature images P6 and P7;

[0124] S2, convolve C5 to obtain C5'' with the same number of channels as C4, add C5'' and C4 to obtain P4', and then convolve P4' to obtain the fused feature image P4;

[0125] S3, upsample C4 once to obtain a feature image C4' with the same resolution as C3, perform a convolution operation on the feature image C4' to obtain C4'' with the same number of channels as C3, add C4'' and C3 to obtain P3', and then perform a convolution operation on P3' to obtain a fused feature image P3;

[0126] Among them, the convolution kernel size of the above convolution can be flexibly set, and this application does not impose any limitation on this.

[0127] Furthermore, the characteristic image C5 is the image with the lowest resolution in a group of initial characteristic images. The above-mentioned second group of fused characteristic images can perform two downsampling operations on the characteristic image C5 to obtain characteristic image P6 and characteristic image P7. The characteristic image P6 and characteristic image P7 are the second group of fused characteristic images. Then, the fused characteristic image P3, the fused characteristic image P4, the fused characteristic image P5, the characteristic image P6 and the characteristic image P7 are determined as the above-mentioned target image data.

[0128] As an optional solution, before inputting the target image data into the target detection model to obtain the target detection result, the method further includes:

[0129] Acquire sample image data, wherein the sample image data is pre-labeled with position information of a sample minimum circumscribed rectangle associated with a sample object, a sample object category corresponding to the sample minimum circumscribed rectangle, a sample orientation category, and a sample rotation angle, wherein the sample rotation angle represents a rotation angle when the horizontal axis in the image coordinate system is rotated to be parallel to the wide side of the sample minimum circumscribed rectangle or a rotation angle when the vertical axis in the image coordinate system is rotated to be parallel to the long side of the sample minimum circumscribed rectangle; perform N rounds of training on an initial detection model according to the sample image data to obtain the target detection model, wherein the model parameters of the target detection model are the model parameters used in the i-th round of training of the initial detection model in the N rounds of training, and the i-th round of training represents the training round in which the model indicators of the initial detection model meet preset conditions, and each round of training determines a target loss value of the initial detection model, and adjusts the model parameters of the initial detection model based on the target loss value.

[0130] Optionally, in an embodiment of the present application, the above-mentioned sample image data may include but are not limited to portrait images, pet images, equipment images, landscape images, etc., and the above-mentioned sample objects may include but are not limited to portraits, pets, equipment, vehicles, buildings, etc.

[0131] It should be noted that the feature extraction and fusion modules and the target detection model can be trained together. The training here is joint training, that is, the model parameters at the feature extraction, fusion and detection heads can be trained together through the loss function.

[0132] Exemplarily, a camera is set at a traffic intersection, and the sample image data is acquired through the camera. The sample objects may include but are not limited to vehicles, pedestrians, traffic signs, etc.

[0133] In an exemplary embodiment, the annotation information of the sample image data may include, but is not limited to, the following:

[0134] S1, location information of the minimum enclosing rectangle of the sample: the horizontal coordinates, vertical coordinates, width, height, etc. of the vertices of the minimum enclosing rectangle. The vertices here may include but are not limited to the center point, upper left corner vertex, lower left corner vertex, upper right corner vertex, lower right corner vertex, etc. of the minimum enclosing rectangle;

[0135] S2, the sample object category corresponding to the sample minimum circumscribed rectangle: for example, cars, pedestrians, traffic signs, etc.;

[0136] S3, sample orientation category: for example, orientation category 0, orientation category 1, orientation category 2, orientation category 3, etc., which are associated with the above-mentioned sample rotation angles. For example, when the X-axis of the image coordinate system is rotated to be parallel to the long side of the minimum circumscribed rectangle of the sample object, it is the same as the sample movement direction after stopping the rotation, which is the 0 orientation category; when the X-axis of the image coordinate system is rotated to be parallel to the long side of the minimum circumscribed rectangle of the sample object, it is positively perpendicular to the sample movement direction after stopping the rotation, and the angle is 90 degrees, which is the 1 orientation category; when the X-axis of the image coordinate system is rotated to be parallel to the long side of the minimum circumscribed rectangle of the sample object, it is opposite to the sample movement direction after stopping the rotation, and the angle is 180 degrees, which is the 2 orientation category; when the X-axis of the image coordinate system is rotated to be parallel to the long side of the minimum circumscribed rectangle of the sample object, it is 270 degrees positively perpendicular to the sample movement direction after stopping the rotation, which is the 3 orientation category.

[0137] Optionally, in an embodiment of the present application, the value of the above N can be flexibly set, and the present application does not impose any limitation on this. The above model parameters may include but are not limited to parameters such as adjustable weights and biases in the initial detection model. During N rounds of training, the model parameters used in each round of training are the model parameters obtained in the previous round of training, and the performance of the model can be improved by continuously adjusting these parameters.

[0138] Optionally, in an embodiment of the present application, the above-mentioned preset conditions can be understood as target indicators set during the training process, such as accuracy, recall rate, etc. When the initial detection model reaches the preset accuracy and recall rate in a certain round of training, it indicates that the training is completed, and the trained initial detection model can be determined as the target detection model. It can be understood that during the training process of the initial detection model, the above-mentioned target error value can be calculated by the loss function to optimize the performance of the model. In each round of training, the model parameters can be adjusted according to the target loss value so that the model can better fit the sample data and improve the detection accuracy and generalization ability.

[0139] For example, Fig.10 is a schematic diagram of another optional object detection method according to an embodiment of the present application, the training process of the initial detection model is as follows Fig.10 As shown:

[0140] S1002, start;

[0141] S1004, collecting and labeling sample image data, including but not limited to performing image denoising, image enhancement, image grayscale conversion, image normalization and other pre-processing on the sample image data before labeling;

[0142] S1006, dividing the data set into a training set, a validation set, and a test set according to a 7:2:1 distribution;

[0143] S1008, using the training set to train the initial detection network, and executing S1010 after the training is completed;

[0144] S1010, using the validation set to verify whether the indicators of the initial detection model meet the preset conditions, and using the test set to test the initial detection model to obtain the target detection model;

[0145] S1012, saving and exporting the target detection model, including but not limited to exporting it as an onnx model;

[0146] S1014, end.

[0147] As an optional scheme, the method of performing N rounds of training on the initial detection model according to the sample image data to obtain the target detection model includes: performing N rounds of training on the initial detection model in the following manner, wherein each round of training includes the following steps: obtaining a predetermined number of sample anchor frames, wherein the number of sample anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the sample image data; detecting the target image data according to a predetermined total number of sample object categories and the number of sample anchor frames, and determining multiple sample prediction probabilities, wherein one sample prediction probability among the multiple sample prediction probabilities is used to represent a prediction probability that an anchor frame at a sampling point belongs to a prediction object category; determining a first loss value of the initial detection model based on the sample object category and the multiple sample prediction probabilities, and adjusting the model parameters of the initial detection model based on the first loss value, wherein the target loss value includes the first loss value.

[0148] Optionally, in an embodiment of the present application, the number of the above sample anchor frames can be flexibly set, and the present application does not impose any limitation on this.

[0149] Exemplarily, several sample image data are input in each batch, each of which includes several of the above-mentioned sampling points, and each sampling point can have the above-mentioned number of sample anchor frames. Taking sampling point A as an example, there are 9 anchor frames in total, among which, the sample prediction probability of an anchor frame A belonging to object category A is 1, then the anchor frame A corresponding to sampling point A belongs to object category A as a positive sample, and the sample prediction probability of anchor frame A belonging to other object categories except object category A is 0, then the anchor frame A corresponding to sampling point A belongs to other object categories except object category A as a negative sample.

[0150] Exemplarily, it may include but is not limited to determining the above-mentioned first loss value of the above-mentioned initial detection model based on the above-mentioned sample object category and the above-mentioned multiple sample prediction probabilities to adjust the model parameters of the initial detection model. Specifically, it may include but is not limited to using the Focal loss loss function to calculate the above-mentioned first loss value, and the first loss value is used to determine the above-mentioned target loss value.

[0151] As an optional solution, the method of training the initial detection model for N rounds according to the sample image data to obtain the target detection model includes: training the initial detection model for N rounds in the following manner, wherein each round of training includes the following steps: detecting the sample image data to determine the predicted coordinate information and predicted size information of the predicted minimum enclosing rectangle; generating a predicted offset for the predicted coordinate information and the predicted size information; determining a predicted rotation angle according to the sample minimum enclosing rectangle and the predicted minimum enclosing rectangle, wherein the predicted rotation angle represents the rotation angle of the sample minimum enclosing rectangle relative to the predicted minimum enclosing rectangle; determining the category indicated by the angle between the orientation vector of the sample object and the long side of the predicted minimum enclosing rectangle as the predicted orientation category; determining a second loss value according to the predicted offset, the predicted rotation angle, the sample offset and the sample rotation angle, determining a third loss value according to the predicted orientation category and the sample orientation category, and adjusting the model parameters of the initial detection model based on the second loss value and the third loss value, wherein the target loss value includes the second loss value and the third loss value.

[0152] Optionally, in an embodiment of the present application, the above-mentioned predicted coordinate information may include but is not limited to the horizontal and vertical coordinates, width, height, etc. of the center point of the above-mentioned predicted minimum enclosing rectangle, and the above-mentioned preset size information can be understood as the size of the anchor box (that is, the predicted minimum enclosing rectangle) in the above-mentioned positive sample, including but not limited to the horizontal and vertical coordinates, width, height, etc. of the center point of the predicted minimum enclosing rectangle, and the above-mentioned predicted offset is obtained by comparing the predicted minimum enclosing rectangle and the sample minimum enclosing rectangle.

[0153] Exemplarily, the horizontal coordinate of the center point of the predicted minimum enclosing rectangle is offset to the right by 10 pixels relative to the horizontal coordinate of the center point of the sample minimum enclosing rectangle, and the vertical coordinate of the center point of the predicted minimum enclosing rectangle is offset upward by 10 pixels relative to the vertical coordinate of the center point of the sample minimum enclosing rectangle, etc. The predicted minimum enclosing rectangle needs to be rotated 30 degrees clockwise to have the same position direction as the above-mentioned predicted minimum enclosing rectangle, that is, the above-mentioned predicted rotation angle is 30 degrees.

[0154] Furthermore, the category of the angle between the orientation vector of the sample object and the long side of the predicted minimum circumscribed rectangle can be determined as the predicted orientation category. For example, the angle category of the orientation vector of the sample object relative to the long side of the predicted minimum circumscribed rectangle is 180 degrees, which can be mapped to orientation category 1, indicating that when the X-axis is rotated to be parallel to the long side of the predicted minimum circumscribed rectangle of the sample object, the orientation after stopping the rotation is 180 degrees from the orientation of the sample object.

[0155] Exemplarily, it may include but is not limited to determining the second loss value of the above-mentioned initial detection model based on the above-mentioned predicted offset, predicted rotation angle, sample offset and sample rotation angle to adjust the model parameters of the initial detection model. Specifically, it may include but is not limited to using the L1 Loss loss function to calculate the above-mentioned second loss value, and the second loss value is used to determine the above-mentioned target loss value; it may also include but is not limited to determining the third loss value of the above-mentioned initial detection model based on the above-mentioned predicted orientation category and sample orientation category to adjust the model parameters of the initial detection model. Specifically, it may include but is not limited to using the cross-loss entropy loss function to calculate the above-mentioned third loss value, and the third loss value is used to determine the above-mentioned target loss value.

[0156] In an exemplary embodiment, the target loss value may be calculated in the following manner, including but not limited to: ,in, represents the first loss value, represents the second loss value, represents the third loss value, is the number of positive samples of the anchor box in a training batch, i represents all samples, including positive samples and negative samples, j is a positive sample, the positive sample needs to calculate the first loss value, the second loss value and the third loss value, while the negative sample only needs to calculate the first loss value.

[0157] As an optional scheme, the determining of the second loss value based on the predicted offset, the predicted rotation angle, the sample offset and the sample rotation angle includes: performing a target encoding operation on the sample offset to obtain a sample encoding offset; determining the second loss value based on the sample encoding offset and the sample rotation angle together with the predicted offset and the predicted rotation angle; or performing a target decoding operation on the predicted offset to obtain a predicted decoding offset; determining the second loss value based on the sample offset and the sample rotation angle together with the predicted decoding offset and the predicted rotation angle.

[0158] Optionally, in an embodiment of the present application, the above-mentioned target encoding operation refers to converting the sample offset to obtain the encoded sample offset for comparison and matching when calculating the loss value, and the above-mentioned target decoding operation can be understood as the inverse operation of the target encoding operation, that is, converting the predicted offset to obtain the decoded predicted offset.

[0159] In an exemplary embodiment, the sample offset is first encoded to obtain a sample coding offset, and then the sample coding offset and the sample rotation angle are processed with the predicted offset and the predicted rotation angle using an L1 Loss function to obtain a second loss value.

[0160] In another exemplary embodiment, the prediction offset is first decoded to obtain the prediction decoding offset, and then the sample decoding offset and the sample rotation angle are processed with the prediction decoding offset and the prediction rotation angle using the L1 Loss loss function to obtain a second loss value, specifically:

[0161] The coded prediction offset is expressed as ( ) as an example, the decoding operation is performed according to the following formula:

[0162]

[0163] in, To predict the center horizontal coordinate, vertical coordinate, width and height of the minimum bounding rectangle, The center horizontal coordinate, vertical coordinate, width and height of the predicted decoding offset, where: express The exponential power of e, express The exponential power of e.

[0164] Exemplarily, the object detection method proposed in the present application can be applied to the common scene of object orientation detection, specifically, including but not limited to:

[0165] S1, training phase:

[0166] S1-a, training image data collection and annotation, collect a certain amount of training data containing the target category to be detected (the above-mentioned sample image data), assume that data of three categories A, B, and C are detected (the number of categories can be flexibly set, for example, the number of categories is not more than 30), the training data is not less than 1000, the number of targets to be detected in the data is not less than 2000 in each category, and each data is manually annotated to obtain the corresponding annotation information of each image, corresponding to a training image, if the training image contains the target category to be detected, assume that the target category is A and B, and there are 2 A objects and 3 B objects, then the corresponding annotation information of the image is processed as a data with five rows and seven columns, where each row represents the annotation information of the object, namely cls, x, y, w, h, theta, o_cls, where cls is the category information to which the object belongs, and it is mapped to 0, 1, and 2 corresponding to A, B, and C (if the detection is redundant or insufficient The three categories can be deduced in turn); x, y are the horizontal and vertical coordinates of the center point of the minimum enclosing rectangle of the target, w, h are the width and height of the minimum enclosing rectangle of the target, and the width is greater than or equal to the height, where x, y, w, h are all normalized values ​​based on the length and width of the original image, theta is the angle that needs to be rotated when the x-axis is rotated to be parallel to the width of the minimum enclosing rectangle of the target in the image coordinate system, and its value is [-π / 2,π / 2), negative numbers represent counterclockwise rotation, and positive numbers represent clockwise rotation, o_cls is one of 0, 1, 2, and 3, indicating the angle category of the orientation vector of the target relative to the long (wide) side of the minimum enclosing rectangle, and its category can be 0 degrees, 90 degrees, 180 degrees, and 270 degrees (tangent to the X-axis). Therefore, it can be mapped to 0, 1, 2, and 3. For example, 0 indicates that the orientation of the target and the inclination angle of the long side of the minimum enclosing rectangle are in the same direction. It should be noted that if the number of sample objects in the sample image data is 0, the content of the corresponding annotation file is also empty.

[0167] S1-b, data preprocessing and data set division:

[0168] S1-b-1, divide the dataset offline before model training: Based on the dataset obtained in the training image data collection and annotation stage, divide the dataset into training set, validation set and test set according to the distribution of 7:2:1.

[0169] S1-c, target orientation model training: The target orientation model is constructed, which may include but is not limited to using an improved retinanet as the target orientation model network. The construction of the orientation model includes but is not limited to the detection network retinanet in this article. The above initial detection model is divided into three parts: a backbone network, a feature pyramid network and a detection network. Among them, the backbone network adopts resnet50, and the C3, C4, and C5 output feature maps of resnet50 are obtained. C3, C4, and C5 (the above set of initial feature images) represent the output results of the network at different stages of feature extraction. The number after C represents how many times the feature map is downsampled relative to the original network input size. For example, C3 is downsampled 3 times, and the resolution of this feature map is 1 / 8 (1 / 23) of the network input. The feature pyramid network takes C3, C4, and C5 as input, and obtains the feature fusion output P3, P4, and P5 through the convolution network, upsampling, and addition operations; at the same time, a convolution with a stride of 2 is added after C5 to obtain the output of P6 and P7, and P3, P4, P5, P6, and P7 are used as the input of the detection network. The detection network shares parameters (model parameters of network training), and the output of the detection network is divided into three branches, one branch is the target category classification branch, one branch is the detection regression branch, and one branch is the target orientation category classification branch.

[0170] Specifically, the output of the target category classification branch is N×C1×H×W, where N is the size of batchsize (the number of sample image data for training the current batch), that is, the number of pictures for each training model, C1 is anchor×class_num, where class_num is the number of target categories to be detected in the data set, anchor is the number of anchor boxes preset by retinanet, for example, the number of anchor boxes is set to 9, H, W are the resolutions of the feature maps output by the detection network, and their values ​​are 1 / 8, 1 / 16 / , 1 / 32, 1 / 64, 1 / 128 of the network input, corresponding to the output sizes of P3, P4, P5, P6, and P7 after the target category classification branch; the output of the detection regression branch is N×C2×H×W, N, H, and W are the same as the target category classification branch, and the size of C2 is anchor×(4+1), where 4 in anchor×(4+1) represents the detected rotated rectangle (minimum enclosing rectangle) relative to its corresponding anchor ( ), as follows:

[0171]

[0172] in The horizontal and vertical coordinates and width and height of the corresponding anchor center. , , is the detected minimum circumscribed rotation rectangle. refers to The exponential power of e, refers to The exponential power of e. The 1 in anchor×(4+1) represents the rotation angle of the detected rotated rectangle (target minimum enclosing rectangle) relative to its corresponding anchor, and its value is [-π / 2,π / 2). The default anchor is a regular rectangle (rotation angle is 0); for the target orientation classification branch, its output is N×X4×H×W, N, H, W are the same as the target category classification branch, C3 is anchor×4, and the 4 here is the angle category of the target's orientation vector relative to the long side of the minimum enclosing rectangle, a total of 4 (0, 1, 2, 3 above), the calculation formula of the model's loss function is as follows:

[0173] in, is the target category classification loss, For each prediction result, it represents all samples, including positive samples and negative samples, j is a positive sample, N pos is the number of positive samples of the prediction box in a training batch, regardless of positive or negative. is the regression loss, is the number of positive samples in a batch size, j is each positive sample, The target orientation classification loss may include but is not limited to using Focal loss to calculate the target category classification loss; using L1 Loss to calculate the detection regression loss L reg , the target orientation classification loss is calculated using the cross entropy loss.

[0174] S1-d, training and validation of the model, initializing the network and network parameters, setting some hyperparameters of the learning network, such as the number of training rounds, learning rate, optimizer, learning strategy, and data enhancement strategy. The data of the training set is sent to the network model in batches according to the batch size, the loss is calculated, the gradient is back-propagated, and the model parameters are updated. At the same time, after each training round, that is, after the data of the training set has been learned by the network, the map index on the validation set is calculated once, and the training parameters of the round with the highest index on the validation set are retained during the training process.

[0175] S2, inference stage, collects inference image data in real time, exports the model parameters and models saved in the training stage as an onnx model, and preprocesses the inference image data collected in real time, such as scaling and normalization, and sends them to the onnx model in batches for inference to obtain an encoding result (the above-mentioned target offset), and performs post-processing, that is, decoding the output of the detection network and performing non-maximum suppression to obtain the output result of the network.

[0176] Through the embodiments of the present application, an end-to-end method for detecting the position, rotation angle and orientation information of an object is implemented, which effectively solves the technical problem of low accuracy in detecting the orientation of objects in autonomous driving, remote sensing aerial photography, and industrial scenes, and further improves the accuracy of the orientation of the object; the orientation of the object is established on the rotation angle of the object, and the rotation angle of the object and the orientation information of the object are controllably output.

[0177] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0178] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0179] According to another aspect of the embodiments of the present application, an object detection device for implementing the above object detection method is also provided. Fig.11 As shown, the device comprises:

[0180] An acquisition module 1102 is used to acquire initial image data to be detected, wherein the initial image data includes a target object to be identified;

[0181] The feature extraction and fusion module 1104 is used to perform a sampling operation on the initial image data to obtain target image data, wherein the resolution of the target image data is smaller than the resolution of the initial image data;

[0182] The detection module 1106 is used to input the target image data into the target detection model to obtain the target detection result, wherein the target detection result includes a target detection frame, a target object category, and a target orientation category. The target detection model is used to predict the target image data and determine the target offset and the target rotation angle. The target detection frame is used to indicate the image area where the target object exists in the target image data. The target object category is used to indicate the object category of the target object. The target orientation category is used to indicate the moving direction of the target object in the image coordinate system. The target offset is used to indicate the position of the target detection frame in the image coordinate system. The target rotation angle is used to indicate the angle by which the target detection frame needs to be rotated in the image coordinate system.

[0183] As an optional solution, the above-mentioned device is used to input the above-mentioned target image data into a target detection model in the following manner to obtain a target detection result: obtain a predetermined number of anchor frames, wherein the above-mentioned number of anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the above-mentioned target image data; detect the above-mentioned target image data according to the predetermined total number of object categories and the above-mentioned number of anchor frames, and determine multiple prediction probabilities, wherein one prediction probability among the above-mentioned multiple prediction probabilities is used to indicate that an anchor frame corresponding to a sampling point includes the target object, and the one sampling point belongs to the object category corresponding to the one anchor frame; determine the anchor frame corresponding to the prediction probability with the largest value among the above-mentioned multiple prediction probabilities as the above-mentioned target detection frame, and determine the object category corresponding to the above-mentioned target detection frame as the above-mentioned target object category.

[0184] As an optional solution, the above-mentioned device is used to input the above-mentioned target image data into the target detection model in the following manner to obtain the target detection result: detect the above-mentioned target image data to determine the coordinate information and size information of the minimum enclosing rectangle of the above-mentioned target object; generate a coded offset based on the above-mentioned coordinate information and the above-mentioned size information; perform a target decoding operation on the above-mentioned coded offset to obtain the above-mentioned target offset; determine the above-mentioned target rotation angle based on the above-mentioned minimum enclosing rectangle and the above-mentioned target detection frame, wherein the above-mentioned target rotation angle represents the rotation angle of the above-mentioned minimum enclosing rectangle relative to the above-mentioned target detection frame; determine the category indicated by the angle between the orientation vector of the above-mentioned target object and the long side of the above-mentioned minimum enclosing rectangle as the above-mentioned target orientation category.

[0185] As an optional solution, the above-mentioned device is used to perform feature extraction and fusion operations on the above-mentioned initial image data in the following manner to obtain target image data: perform a downsampling operation on the above-mentioned initial image data to determine a group of initial feature images, wherein the resolutions of each initial feature image in the above-mentioned group of initial feature images are different; perform a feature fusion operation on the above-mentioned group of initial feature images based on a feature pyramid to obtain a first group of fused feature images, wherein a fused feature image in the above-mentioned first group of fused feature images corresponds to an initial feature image in the above-mentioned group of initial feature images; perform the above-mentioned downsampling operation on the target initial feature image to obtain a second group of fused feature images, wherein the above-mentioned target initial feature image represents the image with the lowest resolution in the above-mentioned group of initial feature images; determine the above-mentioned target image data based on the above-mentioned first group of fused feature images and the above-mentioned second group of fused feature images.

[0186] As an optional solution, the above-mentioned device is also used for: inputting the above-mentioned target image data into the target detection model, obtaining sample image data before obtaining the target detection result, wherein the above-mentioned sample image data is pre-marked with the position information of the sample minimum circumscribed rectangle associated with the sample object, the sample object category corresponding to the above-mentioned sample minimum circumscribed rectangle, the sample orientation category and the sample rotation angle, and the above-mentioned sample rotation angle represents the rotation angle when the horizontal axis in the image coordinate system is rotated to be parallel to the wide side of the above-mentioned sample minimum circumscribed rectangle or the rotation angle when the vertical axis in the above-mentioned image coordinate system is rotated to be parallel to the long side of the above-mentioned sample minimum circumscribed rectangle; performing N rounds of training on the initial detection model according to the above-mentioned sample image data to obtain the above-mentioned target detection model, wherein the model parameters of the above-mentioned target detection model are the model parameters used in the i-th round of training of the above-mentioned initial detection model in the above-mentioned N rounds of training, and the above-mentioned i-th round of training represents the training round in which the model indicators of the above-mentioned initial detection model meet the preset conditions, and each round of training process determines the target loss value of the above-mentioned initial detection model, and adjusts the model parameters of the above-mentioned initial detection model based on the above-mentioned target loss value.

[0187] As an optional solution, the above-mentioned device is used to perform N rounds of training on the initial detection model according to the above-mentioned sample image data in the following manner to obtain the above-mentioned target detection model: perform N rounds of training on the initial detection model in the following manner, wherein each round of training includes the following steps: obtain a predetermined number of sample anchor frames, wherein the above-mentioned number of sample anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the above-mentioned sample image data; detect the above-mentioned target image data according to the predetermined total number of sample object categories and the above-mentioned number of sample anchor frames, and determine multiple sample prediction probabilities, wherein one sample prediction probability among the above-mentioned multiple sample prediction probabilities is used to indicate that a sample anchor frame corresponding to a sample sampling point includes the sample object, and the one sample sampling point belongs to the predicted object category corresponding to the one sample anchor frame; determine a first loss value of the above-mentioned initial detection model based on the above-mentioned sample object category and the above-mentioned multiple sample prediction probabilities, and adjust the model parameters of the above-mentioned initial detection model based on the above-mentioned first loss value, wherein the above-mentioned target loss value includes the above-mentioned first loss value.

[0188] As an optional solution, the above-mentioned device is used to perform N rounds of training on the initial detection model according to the above-mentioned sample image data in the following manner to obtain the above-mentioned target detection model: perform N rounds of training on the initial detection model in the following manner, wherein each round of training includes the following steps: detect the above-mentioned sample image data to determine the predicted coordinate information and predicted size information of the predicted minimum enclosing rectangle; generate a predicted offset for the above-mentioned predicted coordinate information and the above-mentioned predicted size information; determine the predicted rotation angle according to the above-mentioned sample minimum enclosing rectangle and the above-mentioned predicted minimum enclosing rectangle, wherein the above-mentioned predicted rotation angle represents the rotation angle of the above-mentioned sample minimum enclosing rectangle relative to the above-mentioned predicted minimum enclosing rectangle; determine the category indicated by the angle between the orientation vector of the above-mentioned sample object and the long side of the above-mentioned predicted minimum enclosing rectangle as the predicted orientation category; determine a second loss value according to the above-mentioned predicted offset, the above-mentioned predicted rotation angle, the above-mentioned sample offset and the above-mentioned sample rotation angle, determine a third loss value according to the above-mentioned predicted orientation category and the above-mentioned sample orientation category, and adjust the model parameters of the above-mentioned initial detection model based on the above-mentioned second loss value and the above-mentioned third loss value, wherein the above-mentioned target loss value includes the above-mentioned second loss value and the above-mentioned third loss value.

[0189] As an optional scheme, the above-mentioned device is used to determine the second loss value according to the above-mentioned prediction offset, the above-mentioned prediction rotation angle, the above-mentioned sample offset and the above-mentioned sample rotation angle in the following manner: performing a target encoding operation on the above-mentioned sample offset to obtain a sample encoding offset; determining the above-mentioned second loss value based on the above-mentioned sample encoding offset and the above-mentioned sample rotation angle together with the above-mentioned prediction offset and the above-mentioned prediction rotation angle; or performing a target decoding operation on the above-mentioned prediction offset to obtain a prediction decoding offset; determining the above-mentioned second loss value based on the above-mentioned sample offset and the above-mentioned sample rotation angle together with the above-mentioned prediction decoding offset and the above-mentioned prediction rotation angle.

[0190] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0191] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0192] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0193] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0194] Fig.12 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0195] It should be noted that Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0196] like Fig.12As shown, the computer system 1200 includes a central processing unit 1201 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1202 (ROM) or the program loaded from the storage part 1208 to the random access memory 1203 (RAM). Various programs and data required for system operation are also stored in the random access memory 1203. The central processing unit 1201, the read-only memory 1202 and the random access memory 1203 are connected to each other through a bus 1204. The input / output interface 1205 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1204.

[0197] The following components are connected to the input / output interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0198] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processor 1201, various functions defined in the system of the present application are executed.

[0199] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processor 1201, various functions provided by the embodiments of the present application are performed.

[0200] According to another aspect of the embodiment of the present application, an electronic device for implementing the above object detection method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.13 As shown, the electronic device includes a memory 1302 and a processor 1304. The memory 1302 stores a computer program, and the processor 1304 is configured to execute the steps in any of the above method embodiments through the computer program.

[0201] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0202] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0203] Alternatively, a person skilled in the art may understand that: Fig.13 The structure shown is for illustration only. Fig.13 The structure of the electronic device is not limited. Fig.13 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.13 Different configurations are shown.

[0204] Among them, the memory 1302 can be used to store software programs and modules, such as program instructions / modules corresponding to the object detection method and device in the embodiments of the present application. The processor 1304 executes various functional applications and data processing by running the software programs and modules stored in the memory 1302, that is, realizing the above-mentioned object detection method. The memory 1302 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1302 may further include a memory remotely located relative to the processor 1304, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1302 can be specifically, but not limited to, used to store information such as target image data. As an example, such as Fig.13 As shown, the memory 1302 may include but is not limited to the acquisition module 1102, feature extraction and fusion module 1104, and detection module 1106 in the object detection device. In addition, it may also include but is not limited to other module units in the object detection device, which will not be repeated in this example.

[0205] Optionally, the transmission device 1306 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 1306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1306 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0206] In addition, the electronic device further includes: a display 1308 for displaying the target detection result; and a connection bus 1310 for connecting various module components in the electronic device.

[0207] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0208] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the object detection method provided in various optional implementations of the above-mentioned object detection aspects.

[0209] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0210] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0211] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0212] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0213] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0214] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0215] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0217] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An object detection method, characterized in that: include: Acquire initial image data to be detected, wherein the initial image data includes a target object to be identified; Performing feature extraction and fusion operations on the initial image data to obtain target image data, wherein the resolution of the target image data is smaller than the resolution of the initial image data; Input the target image data into a target detection model to obtain a target detection result, wherein the target detection result includes a minimum bounding rectangle, a target object category, and a target orientation category. The target detection model is used to predict the target image data and determine a target offset and a target rotation angle. The minimum bounding rectangle is used to indicate an image region in the target image data where the target object exists. The target object category is used to indicate an object category of the target object. The target orientation category is used to indicate a moving direction of the target object in an image coordinate system when the target detection frame is adjusted using the target rotation angle to obtain the minimum bounding rectangle. The target offset is used to indicate a position of the minimum bounding rectangle in the image coordinate system. The target rotation angle indicates a rotation angle of the minimum bounding rectangle relative to the target detection frame. The method further includes: processing the target image data using a target orientation category classification branch of the target detection model, and directly outputting the target orientation category by the target orientation category classification branch, wherein the target orientation category represents an orientation category determined from preset orientation categories, and the preset orientation category is a coordinate axis of the image coordinate system of the rotated sample image in the training phase of the target detection model, and when the coordinate axis is rotated to be parallel to the long side of the sample minimum circumscribed rectangle corresponding to the sample object in the sample image, the orientation category is determined according to the angle category between the coordinate axis and the moving direction of the sample object; The method also includes: using the detection regression branch of the target detection model to process the target image data to obtain the target rotation angle and the encoding offset, adjusting the target detection frame according to the target rotation angle and the encoding offset to obtain the minimum circumscribed rectangle, wherein the encoding offset is used to adjust the size and position of the target detection frame, and performing a target decoding operation on the encoding offset to obtain the target offset.

2. The method according to claim 1, characterized in that The step of inputting the target image data into a target detection model to obtain a target detection result includes: Acquire a predetermined number of anchor frames, wherein the number of anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the target image data; Detect the target image data according to a predetermined total number of object categories and the number of anchor boxes, and determine a plurality of prediction probabilities, wherein a prediction probability among the plurality of prediction probabilities is used to indicate a prediction probability that an anchor box corresponding to a sampling point includes the target object, and that the sampling point belongs to the object category corresponding to the anchor box; The anchor frame corresponding to the predicted probability with the largest value among the multiple predicted probabilities is determined as the target detection frame, and the object category corresponding to the target detection frame is determined as the target object category.

3. The method according to claim 1, characterized in that The step of inputting the target image data into a target detection model to obtain a target detection result includes: Detecting the target image data to determine coordinate information and size information of the minimum circumscribed rectangle of the target object; Generate the encoding offset according to the coordinate information and the size information; The target decoding operation is performed on the encoded offset to obtain the target offset.

4. The method according to claim 1, characterized in that: The performing feature extraction and fusion operations on the initial image data to obtain target image data includes: Performing a downsampling operation on the initial image data to determine a set of initial feature images, wherein the resolutions of the initial feature images in the set of initial feature images are different; Performing a feature fusion operation on the set of initial feature images based on the feature pyramid to obtain a first set of fused feature images, wherein a fused feature image in the first set of fused feature images corresponds to an initial feature image in the set of initial feature images; Performing the downsampling operation on the target initial feature image to obtain a second set of fused feature images, wherein the target initial feature image represents an image with the lowest resolution in the set of initial feature images; The target image data is determined according to the first group of fused feature images and the second group of fused feature images.

5. The method according to claim 1, characterized in that Before inputting the target image data into the target detection model to obtain the target detection result, the method further includes: Acquire sample image data, wherein the sample image data is pre-labeled with position information of the sample minimum circumscribed rectangle associated with the sample object, the sample object category corresponding to the sample minimum circumscribed rectangle, the sample orientation category, and the sample rotation angle, wherein the sample rotation angle represents the rotation angle when the horizontal axis in the image coordinate system is rotated to be parallel to the wide side of the sample minimum circumscribed rectangle or the rotation angle when the vertical axis in the image coordinate system is rotated to be parallel to the long side of the sample minimum circumscribed rectangle; The initial detection model is trained N rounds according to the sample image data to obtain the target detection model, wherein the model parameters of the target detection model are the model parameters used in the i-th round of training of the initial detection model in the N rounds of training, and the i-th round of training represents the training round in which the model indicators of the initial detection model meet preset conditions, and each round of training determines the target loss value of the initial detection model, and adjusts the model parameters of the initial detection model based on the target loss value.

6. The method according to claim 5, characterized in that The performing N rounds of training on the initial detection model according to the sample image data to obtain the target detection model comprises: The initial detection model is trained for N rounds in the following manner, where each round of training includes the following steps: Acquire a predetermined number of sample anchor frames, wherein the number of sample anchor frames is used to indicate the number of anchor frames used for sampling each sampling point in the sample image data; Detecting the target image data according to a predetermined total number of sample object categories and the number of sample anchor frames, and determining a plurality of sample prediction probabilities, wherein a sample prediction probability among the plurality of sample prediction probabilities is used to indicate that a sample anchor frame corresponding to a sample sampling point includes the sample object, and a prediction probability that the sample sampling point belongs to a prediction object category corresponding to the sample anchor frame; A first loss value of the initial detection model is determined based on the sample object category and the multiple sample prediction probabilities, and a model parameter of the initial detection model is adjusted based on the first loss value, wherein the target loss value includes the first loss value.

7. The method according to claim 5, characterized in that The performing N rounds of training on the initial detection model according to the sample image data to obtain the target detection model comprises: The initial detection model is trained for N rounds in the following manner, where each round of training includes the following steps: Detecting the sample image data to determine predicted coordinate information and predicted size information of a predicted minimum circumscribed rectangle; generating a predicted offset for the predicted coordinate information and the predicted size information; Determining a predicted rotation angle according to the sample minimum enclosing rectangle and the predicted minimum enclosing rectangle, wherein the predicted rotation angle represents a rotation angle of the sample minimum enclosing rectangle relative to the predicted minimum enclosing rectangle; Determine the category indicated by the angle between the orientation vector of the sample object and the long side of the predicted minimum circumscribed rectangle as the predicted orientation category; A second loss value is determined according to the predicted offset, the predicted rotation angle, the sample offset and the sample rotation angle, a third loss value is determined according to the predicted orientation category and the sample orientation category, and model parameters of the initial detection model are adjusted based on the second loss value and the third loss value, wherein the target loss value includes the second loss value and the third loss value.

8. The method according to claim 7, characterized in that The determining of the second loss value according to the predicted offset, the predicted rotation angle, the sample offset and the sample rotation angle comprises: Perform a target encoding operation on the sample offset to obtain a sample coding offset; determine the second loss value based on the sample coding offset and the sample rotation angle together with the prediction offset and the prediction rotation angle; or A target decoding operation is performed on the predicted offset to obtain a predicted decoding offset; and the second loss value is determined based on the sample offset and the sample rotation angle together with the predicted decoding offset and the predicted rotation angle.

9. An object detection device, characterized in that: include: An acquisition module, used for acquiring initial image data to be detected, wherein the initial image data includes a target object to be identified; A feature extraction and fusion module, used to perform feature extraction and fusion operations on the initial image data to obtain target image data, wherein the resolution of the target image data is smaller than the resolution of the initial image data; a detection module, configured to input the target image data into a target detection model to obtain a target detection result, wherein the target detection result includes a minimum bounding rectangle, a target object category, and a target orientation category; the target detection model is configured to predict the target image data, determine a target offset and a target rotation angle; the minimum bounding rectangle is configured to indicate an image region in the target image data where the target object exists; the target object category is configured to indicate an object category of the target object; the target orientation category is configured to indicate a moving direction of the target object in an image coordinate system when the target detection frame is adjusted using the target rotation angle to obtain the minimum bounding rectangle; the target offset is configured to indicate a position of the minimum bounding rectangle in the image coordinate system; and the target rotation angle indicates a rotation angle of the minimum bounding rectangle relative to the target detection frame; The device is also used for: processing the target image data by using the target orientation category classification branch of the target detection model, and directly outputting the target orientation category by the target orientation category classification branch, wherein the target orientation category represents an orientation category determined from preset orientation categories, and the preset orientation category is the coordinate axis of the image coordinate system of the rotated sample image in the training stage of the target detection model, and when the coordinate axis is rotated to be parallel to the long side of the sample minimum circumscribed rectangle corresponding to the sample object in the sample image, the orientation category is determined according to the angle category between the coordinate axis and the moving direction of the sample object; The device is also used to: use the detection regression branch of the target detection model to process the target image data to obtain the target rotation angle and the encoding offset, adjust the target detection frame according to the target rotation angle and the encoding offset to obtain the minimum circumscribed rectangle, wherein the encoding offset is used to adjust the size and position of the target detection frame, and perform a target decoding operation on the encoding offset to obtain the target offset.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images