Target detection method and device, equipment and medium

Abstract

The embodiment of the invention provides a target detection method and device, equipment and a medium, and the method specifically comprises the steps: carrying out the target detection of a to-be-detected image through a target detection model; a backbone network in the target detection model comprises at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module; the first long sequence processing module comprises a first branch, a second branch, a fusion module and a third linear processing module; the first branch comprises a first linear processing module, a convolution processing module, a first activation module and a selective processing module; the selective processing module comprises a plurality of processing layers, each processing layer corresponds to one state matrix, and the state matrix of the (t-1) th processing layer is updated according to the zero-order keeping result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the tth processing layer. According to the embodiment of the invention, the precision and robustness of target detection can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular, to a target detection method, device, equipment and medium. Background Art

[0002] Object detection is an important application of computer vision technology, which is used to detect objects of preset categories (such as people, buildings or cars, etc.) in images. Object detection has been widely used in scenarios such as home security, surveillance security, autonomous driving, and traffic monitoring.

[0003] The current object detection method usually first extracts the feature representation of the image to be detected, and then determines the category label and bounding box of the target object in the image to be detected based on the feature representation. In the related art, CNN (Convolutional Neural Network) is usually used to extract the feature representation of the image to be detected.

[0004] Although convolutional neural networks have significant advantages in extracting local features, their convolution operations can only perceive the features of local areas at each layer, making it difficult to capture long-distance dependencies. The lack of such long-distance dependencies can easily cause the positional relationship information between the target object and other objects in the image to be detected to be omitted in the feature extraction results, thereby increasing the possibility of errors in the bounding box of the target object. Summary of the invention

[0005] The embodiment of the present application provides a target detection method, which can improve the accuracy and robustness of target detection.

[0006] Correspondingly, an embodiment of the present application also provides a target detection device, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, an embodiment of the present application discloses a target detection method, which includes:

[0008] Receiving an image to be detected;

[0009] Using the target detection model, performing target detection on the image to be detected to obtain a corresponding target detection result;

[0010] The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module;

[0011] The first long sequence processing module includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result;

[0012] The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using a first weight matrix, linearly processing the input matrix to obtain a first matrix; using a zero-order hold technique, processing the first matrix to obtain a second matrix; updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the t-th processing layer; obtaining the output matrix of the t-th processing layer according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t is a positive integer;

[0013] The method of using the target detection model to perform target detection on the image to be detected includes:

[0014] The backbone network determines a plurality of first image features corresponding to the image to be detected;

[0015] The neck network performs feature fusion processing according to the multiple first image features to obtain multiple second image features;

[0016] The detection network determines the target detection result according to the multiple second image features.

[0017] In order to solve the above problems, an embodiment of the present application discloses a target detection device, the device comprising:

[0018] A receiving module, used for receiving an image to be detected;

[0019] The target detection module is used to perform target detection on the image to be detected by using the target detection model to obtain the corresponding target detection result;

[0020] The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module;

[0021] The first long sequence processing module includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result;

[0022] The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using a first weight matrix, linearly processing the input matrix to obtain a first matrix; using a zero-order hold technique, processing the first matrix to obtain a second matrix; updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the t-th processing layer; obtaining the output matrix of the t-th processing layer according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t is a positive integer;

[0023] The target detection module comprises:

[0024] A feature extraction module, used to determine a plurality of first image features corresponding to the image to be detected by using the backbone network;

[0025] A feature fusion module, used to utilize the neck network to perform feature fusion processing according to the plurality of first image features to obtain a plurality of second image features;

[0026] The result determination module is used to determine the target detection result according to the plurality of second image features by using the detection network.

[0027] Optionally, updating the first weight matrix and the second weight matrix according to the input matrix includes:

[0028] Determine the statistics corresponding to the dimensional features in the input matrix;

[0029] Determine the target dimension characteristics that the statistics meet the preset conditions;

[0030] For the rows or columns corresponding to the target dimensional features in the weight matrix and the second weight matrix, the corresponding weight values ​​are increased.

[0031] Optionally, the statistic includes: variance or covariance, and the preset condition includes: the statistic is greater than a threshold.

[0032] Optionally, updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix includes:

[0033] Multiply the second matrix and the input matrix to obtain a first product result;

[0034] Multiplying the zero-order hold result of the second weight matrix by the state matrix of the t-1th processing layer to obtain a second product result;

[0035] The first product result and the second product result are added to obtain the state matrix of the tth processing layer.

[0036] Optionally, the backbone network includes: a first convolution module, a second convolution module, a first long sequence processing module A, a third convolution module, a first long sequence processing module B, a fourth convolution module, a first long sequence processing module C, a fifth convolution module, a first long sequence processing module D and a spatial pyramid pooling module connected in sequence;

[0037] Among them, the first long sequence processing module B outputs the first image feature A to the neck network; the first long sequence processing module C outputs the first image feature B to the neck network; and the spatial pyramid pooling module outputs the first image feature C to the neck network.

[0038] Optionally, the neck network includes: a first-level feature fusion module, a second-level feature fusion module and a third-level feature fusion module;

[0039] The first-level feature fusion module includes: a sixth convolution module, a first upsampling module, a first splicing module, a second long sequence processing module A, a seventh convolution module, a second upsampling module, a second splicing module, a second long sequence processing module B and a tenth convolution module connected in series in sequence;

[0040] The second-level feature fusion module includes: an eighth convolution module, a third splicing module, a second long sequence processing module C, and an eleventh convolution module connected in series in sequence;

[0041] The third-level feature fusion module includes: a ninth convolution module, a fourth splicing module, a second long sequence processing module D and a twelfth convolution module connected in series in sequence.

[0042] Optionally, the spatial pyramid pooling module includes: a thirteenth convolution module, p maximum pooling modules, a fifth splicing module and a fourteenth convolution module; p is a positive integer greater than 1.

[0043] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0044] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.

[0045] The embodiments of the present application include the following advantages:

[0046] In the technical solution of the embodiment of the present application, a long sequence processing model is set in the backbone network of the target detection model. The selective processing module of the long sequence processing model includes: multiple processing layers. And, according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix, the state matrix of the t-1th processing layer is updated to obtain the state matrix of the tth processing layer.

[0047] The update mechanism of the above-mentioned state matrix can effectively integrate past information and current information, so that the target detection model can remember the past information that appeared earlier in the image features, and can use this past information to establish long-distance dependencies in subsequent processing. Therefore, the update mechanism of the above-mentioned state matrix can effectively improve the ability of the target detection model to handle long-distance dependencies, so that the target detection model can more accurately identify the target object when facing complex images, and thus can effectively reduce the probability of errors in the bounding box in the target detection results. For example, in an image containing multiple target objects that are mutually occluded or far away, the target detection model can use the established long-distance dependencies to accurately outline the complete outline and position of each target object, reduce false detections and missed detections, and thereby improve the accuracy and robustness of target detection.

[0048] In addition, the convolution module in the backbone network can extract local image features of the image to be detected, so that the first long sequence processing module can mine the long-distance dependencies based on these local image features. The spatial pyramid pooling module in the backbone network can fuse the output image features of the first long sequence processing module with other image features at different scales. In this way, the fused image features contain both long-distance dependency information and multi-scale characteristics. Therefore, the expressiveness and robustness of image features can be improved, so that the target detection model can better cope with target detection at different scales and complex scenes, and enhance the target detection model's recognition ability and positioning accuracy of target objects in images. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the steps of a target detection method according to an embodiment of the present application;

[0050] Figure 2 is a schematic diagram of the structure of a target detection model according to an embodiment of the present application;

[0051] Figure 3 It is a schematic diagram of the structure of a backbone network of an embodiment of the present application;

[0052] Figure 4 is a structural diagram of a first long sequence processing module according to an embodiment of the present application;

[0053] Figure 5 It is a structural diagram of a spatial pyramid pooling module according to an embodiment of the present application;

[0054] Figure 6 is a schematic diagram of the structure of a neck network according to an embodiment of the present application;

[0055] Figure 7 is a schematic structural diagram of a target detection device according to an embodiment of the present application;

[0056] Figure 8 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The embodiments of the present application can be applied to target detection scenarios. In target detection scenarios, the target detection model can extract feature representations of the image to be detected, and determine the category label and bounding box of the target object in the image to be detected based on the feature representations.

[0059] The category label is used to identify the specific category of the target object in the image to be detected. For example, if the target object is a pedestrian, the category label can be "pedestrian"; if the target object is a vehicle, the category label can be "vehicle", etc. The category label is an identifier for classifying the target object.

[0060] A bounding box is usually a rectangular box that defines the location of the target object in the image to be detected. The bounding box can be represented by the coordinates of the upper left corner and the coordinates of the lower right corner, or the bounding box can be represented by the center coordinates, width, and height. The purpose of the bounding box is to clearly define the specific location area of ​​the target object in the image.

[0061] The target detection method of the embodiment of the present application can be applied to scenarios such as home security, surveillance security, autonomous driving, and traffic monitoring. In the home security scenario, the target detection method is used to detect abnormal people or objects that invade. In the surveillance security scenario, the target detection method is used to find specific targets with abnormal behaviors (such as wandering, climbing over fences, etc.). In the autonomous driving scenario, the target detection method is used to provide surrounding target information for vehicle driving. In the traffic monitoring scenario, the target detection method is used to detect vehicle type and vehicle location.

[0062] In the embodiments of the present application, the target object refers to an object of a preset category that needs to be detected. For example, in a home security scenario, the target object may be an abnormal intruder or a moving pet, etc. In a surveillance security scenario, the target object may be a suspicious person with abnormal behavior. In an autonomous driving scenario, the target object may be other vehicles, pedestrians, traffic signs, and signal lights around the vehicle. In a traffic monitoring scenario, the target object may be various vehicles and damage to traffic facilities, etc.

[0063] In the related art, convolutional neural networks are usually used to extract feature representations of the image to be detected. Although convolutional neural networks have significant advantages in local feature extraction, their convolution operation can only perceive the features of the local area at each layer, making it difficult to capture long-distance dependencies. The lack of such long-distance dependencies can easily cause the positional relationship information between the target object and other objects in the image to be detected to be omitted in the feature extraction results, thereby increasing the possibility of errors in the bounding box of the target object.

[0064] Taking the detection of urban traffic images as an example, since it is difficult for convolutional neural networks to capture the long-distance dependencies between vehicles and intersection layouts, traffic signs, etc., they cannot make judgments based on the inherent differences in spatial position and traffic rules between the left-turn waiting area where the vehicle should be located and the straight lane. Therefore, vehicles that should be in the left-turn waiting area may be mistakenly located on the straight lane.

[0065] In view of the technical problem that the possibility of errors in the bounding box in the target detection result in the related art increases, the embodiment of the present application provides a target detection method, which specifically includes the following steps:

[0066] Receiving an image to be detected;

[0067] Using the target detection model, target detection is performed on the above-mentioned image to be detected to obtain the corresponding target detection result;

[0068] The target detection model comprises: a backbone network, a neck network and a detection network connected in sequence; the backbone network comprises: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module;

[0069] The first long sequence processing module includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result;

[0070] The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using the first weight matrix, linearly processing the input matrix to obtain the first matrix; using the zero-order hold technology, processing the first matrix to obtain the second matrix; according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix, updating the state matrix of the t-1th processing layer to obtain the state matrix of the t-th processing layer; according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix, obtaining the output matrix of the t-th processing layer; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t is a positive integer;

[0071] The target detection model is used to perform target detection on the image to be detected, including:

[0072] The backbone network determines a plurality of first image features corresponding to the image to be detected;

[0073] The neck network performs feature fusion processing according to the multiple first image features to obtain multiple second image features;

[0074] The detection network determines the target detection result based on the above-mentioned multiple second image features.

[0075] The embodiment of the present application sets a long sequence processing model in the backbone network of the target detection model. The selective processing module of the long sequence processing model includes: multiple processing layers. And, according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix, the state matrix of the t-1th processing layer is updated to obtain the state matrix of the tth processing layer.

[0076] The update mechanism of the above-mentioned state matrix can effectively integrate past information and current information, so that the target detection model can remember the past information that appeared earlier in the image features, and can use this past information to establish long-distance dependencies in subsequent processing. Therefore, the update mechanism of the above-mentioned state matrix can effectively improve the ability of the target detection model to handle long-distance dependencies, so that the target detection model can more accurately identify the target object when facing complex images, and thus can effectively reduce the probability of errors in the bounding box in the target detection results. For example, in an image containing multiple target objects that are mutually occluded or far away, the target detection model can use the established long-distance dependencies to accurately outline the complete outline and position of each target object, reduce false detections and missed detections, and thereby improve the accuracy and robustness of target detection.

[0077] In addition, the convolution module in the backbone network can extract local image features of the image to be detected, so that the first long sequence processing module can mine the long-distance dependencies based on these local image features. The spatial pyramid pooling module in the backbone network can fuse the output image features of the first long sequence processing module with other image features at different scales. In this way, the fused image features contain both long-distance dependency information and multi-scale characteristics. Therefore, the expressiveness and robustness of image features can be improved, so that the target detection model can better cope with target detection at different scales and complex scenes, and enhance the target detection model's recognition ability and positioning accuracy of target objects in images.

[0078] Method Embodiment

[0079] Reference Figure 1 , shows a schematic diagram of the steps of a target detection method according to an embodiment of the present application, which may specifically include the following steps:

[0080] Step 101, receiving an image to be detected;

[0081] Step 102: using the target detection model to perform target detection on the above-mentioned image to be detected to obtain a corresponding target detection result;

[0082] The target detection model comprises: a backbone network, a neck network and a detection network connected in sequence; the backbone network comprises: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module;

[0083] The first long sequence processing module includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result;

[0084] The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using the first weight matrix, linearly processing the input matrix to obtain the first matrix; using the zero-order hold technology, processing the first matrix to obtain the second matrix; updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the t-th processing layer; obtaining the output matrix of the t-th processing layer according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t can be a positive integer;

[0085] The step 102 uses a target detection model to perform target detection on the image to be detected, including:

[0086] Step 121, the backbone network determines a plurality of first image features corresponding to the image to be detected;

[0087] Step 122: The neck network performs feature fusion processing according to the multiple first image features to obtain multiple second image features;

[0088] Step 123: The detection network determines the target detection result according to the above-mentioned multiple second image features.

[0089] Figure 1 The illustrated method embodiment can be applied to an electronic device. The electronic device can be a smart device, for example, the smart device can be a smart door lock, a camera, a surveillance camera, or other electronic device with a shooting function. The electronic device can also be a computer device that deploys a von Neumann architecture, for example, the computer device can be a desktop computer, a laptop computer, a server, etc.

[0090] In step 101, the image to be detected may come from an image acquisition device such as a camera, a monitoring device, etc. The image acquisition device may acquire the image to be detected at a frame rate such as 24 frames per second, 30 frames per second, etc., and send the image to be detected to the electronic device. It is understood that the embodiments of the present application do not limit the specific image to be detected and the specific source of the image to be detected.

[0091] In step 102, the image to be detected may be input into a target detection model to obtain a target detection result output by the target detection model.

[0092] Reference Figure 2 , shows a schematic diagram of the structure of a target detection model of an embodiment of the present application, which specifically includes: a backbone network 201, a neck network 202 and a detection network 203 connected in sequence.

[0093] The backbone network 201 is used to extract features from the input image to obtain a plurality of first image features. In the training phase, the input image may be an image sample. In the image recognition phase, the input image may be an image to be detected.

[0094] The neck network 202 is used to perform feature fusion processing on the multiple first image features to obtain multiple second image features.

[0095] The detection network 203 is used to determine the target detection result according to the above-mentioned multiple second image features.

[0096] The backbone network specifically includes: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module. The convolution module may be before the first long sequence processing module, and the spatial pyramid pooling module may be after the first long sequence processing module.

[0097] The convolution module in the backbone network can extract local image features from the image to be detected, so that the first long sequence processing module can mine the long-distance dependencies based on these local image features. The spatial pyramid pooling module in the backbone network can fuse the output image features of the first long sequence processing module with other image features at different scales.

[0098] The embodiment of the present application does not limit the specific structure of the convolution module. For example, the convolution module may include: at least one convolution layer, at least one batch normalization layer, and at least one activation function. It is understood that those skilled in the art can adopt the specific structure of the convolution module according to actual application requirements.

[0099] Reference Figure 3, shows a schematic diagram of the structure of a backbone network of an embodiment of the present application, the backbone network specifically includes: a first convolution module 301, a second convolution module 302, a first long sequence processing module A303, a third convolution module 304, a first long sequence processing module B305, a fourth convolution module 306, a first long sequence processing module C307, a fifth convolution module 308, a first long sequence processing module D309 and a spatial pyramid pooling module 310 connected in sequence; the first long sequence processing module A303, the first long sequence processing module B305 and the first long sequence processing module C307 are all specific implementations of the first long sequence processing module, which can use the structure of the first long sequence processing module.

[0100] Among them, the first long sequence processing module B305 outputs the first image feature A to the neck network; the first long sequence processing module C307 outputs the first image feature B to the neck network; and the spatial pyramid pooling module 310 outputs the first image feature C to the neck network.

[0101] The first convolution module 301 performs convolution processing on the image to be detected to obtain a first convolution result.

[0102] The second convolution module 302 performs convolution processing on the first convolution result to obtain a second convolution result.

[0103] The first long sequence processing module A303 performs convolution processing on the second convolution result to obtain the first image feature A.

[0104] The third convolution module 304 performs convolution processing on the first image feature A to obtain a third convolution result.

[0105] The first long sequence processing module B305 performs convolution processing on the third convolution result to obtain a first processing result.

[0106] The fourth convolution module 306 performs convolution processing on the first processing result to obtain a fourth convolution result.

[0107] The first long sequence processing module C307 performs convolution processing on the fourth convolution result to obtain the first image feature B.

[0108] The fifth convolution module 308 performs convolution processing on the first image feature B to obtain a fifth convolution result.

[0109] The first long sequence processing module D309 performs convolution processing on the fifth convolution result to obtain a second processing result.

[0110] The spatial pyramid pooling module 310 performs spatial pyramid pooling processing on the second processing result to obtain the first image feature C.

[0111] Reference Figure 4, showing a schematic diagram of the structure of a first long sequence processing module of an embodiment of the present application, the first long sequence processing module includes: a first branch 401, a second branch 402, a fusion module 403 and a third linear processing module 404; the first branch 401 includes: a first linear processing module 411, a convolution processing module 412, a first activation module 413, and a selective processing module 414; the second branch 402 includes: a second linear processing module 421 and a second activation module 422; the fusion module 403 fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module 404 performs linear processing on the fusion result.

[0112] In practical applications, the convolution module may be before the first long sequence processing module, and the output of the convolution module may be a 4-dimensional matrix. Assuming that the first long sequence processing module is used to process a three-dimensional matrix, the embodiment of the present application may perform a reshape (reconstruction) process on the input of the first long sequence processing module (that is, the output of the convolution module), and the above reshape process is used to convert a four-dimensional matrix into a three-dimensional matrix. For example, if the dimension of the input of the first long sequence processing module is [N, C, H, W], the reshape process can convert the input into a dimension of [N, C, H*W].

[0113] The first linear processing module 411, the second linear processing module 421 and the third linear processing module 404 can be used to perform linear processing on the corresponding input. The fully connected layer can serve as a tool for implementing linear processing. For example, the input (such as the feature map output by the convolution module) is flattened as the input of the fully connected layer, and linear processing is performed through the weight matrix and bias vector of the fully connected layer to obtain the output after linear processing, and the output can be used as the input of the subsequent module.

[0114] The convolution processing module 412 is used to perform convolution operations on the input, extract local features in the image, and enhance the expressiveness of image features. It slides and calculates on the input data through a specific convolution kernel to capture feature information at different positions and provide more discriminative features for subsequent target detection processing. The structure of the convolution processing module 412 can be referred to the convolution module mentioned above, and will not be described in detail here.

[0115] The first activation module 413 is used to perform nonlinear transformation on the features processed by the convolution processing module 412, and enhance the expression ability of the target detection model by introducing nonlinear factors, so that the target detection model can learn and process more complex feature relationships. The first activation module 413 can map the output values ​​after convolution, such as setting values ​​less than zero to zero and values ​​greater than zero to remain unchanged, so as to increase the nonlinearity of the target detection model and help the target detection model better distinguish the target object from the background or the difference between different target objects.

[0116] The selective processing module includes a plurality of processing layers. The number of processing layers can be determined by those skilled in the art according to actual application requirements, for example, the number of processing layers can be 6 or the like.

[0117] Each processing layer corresponds to a state matrix. The state matrix is ​​an important basis for the processing of the processing layer. On the one hand, it participates in the generation of the output matrix of the current processing layer, and on the other hand, it provides a basis for updating the state matrix of the next layer, realizing the orderly flow of information between processing layers and facilitating the efficient operation of the target detection model.

[0118] The processing process of the t-th processing layer includes: using the first weight matrix, linearly processing the input matrix to obtain the first matrix; using the zero-order hold technology to process the first matrix to obtain the second matrix; according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix, the state matrix of the t-1th processing layer is updated to obtain the state matrix of the t-th processing layer; according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix, the output matrix of the t-th processing layer is obtained; wherein the first weight matrix and the second weight matrix are updated according to the input matrix. For the state matrix of the first processing layer, the initial state matrix can be updated according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the first processing layer. wherein the initial state matrix can be determined by those skilled in the art according to actual application requirements. The zero-order hold result of the second weight matrix can be the result obtained by processing the second weight matrix using the zero-order hold technology. The zero-order hold result of the state matrix of the t-th processing layer can be the result obtained by processing the state matrix of the t-th processing layer using the zero-order hold technology.

[0119] By using the first weight matrix to perform linear processing on the input matrix, the image features contained in the input matrix can be re-weighted and integrated to obtain the first matrix. This process can dig out the hidden feature associations in the input matrix. This linear processing process can be achieved through matrix multiplication. The element values ​​in the first weight matrix determine the contribution of the corresponding elements in the input matrix when generating the first matrix, thereby achieving targeted feature extraction and conversion of the input matrix, providing more representative intermediate results for subsequent processing.

[0120] The first matrix is ​​processed using the zero-order hold technique to obtain the second matrix. The purpose is to maintain the values ​​in the first matrix within a certain position range. This helps to retain the key features that are important for processing long-distance dependencies in target detection, so that the second matrix can better reflect these feature information.

[0121] Zero-order hold technology is a commonly used signal sampling and holding technology, which is mainly used to convert continuous-time signals into discrete signals. The basic idea is to keep the input signal unchanged for one cycle until the next sampling value. In the specific implementation, the first matrix can be divided into 3×3 or 5×5 target areas, and the central pixel value in the target area is used as the retention value of the target area.

[0122] The state matrix of the t-1th processing layer is updated according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the tth processing layer.

[0123] The above-mentioned state matrix update mechanism can effectively integrate past information and current information, so that the target detection model can remember the past information that appeared earlier in the image features, and can use this past information to establish long-distance dependencies in subsequent processing.

[0124] The above process of updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix specifically includes: multiplying the second matrix and the input matrix to obtain a first product result; multiplying the zero-order hold result of the second weight matrix and the state matrix of the t-1th processing layer to obtain a second product result; adding the first product result and the second product result to obtain the state matrix of the tth processing layer.

[0125] The embodiment of the present application adds the product result of the second matrix and the input matrix, and the product result of the zero-order hold result of the second weight matrix and the state matrix of the previous layer, so that the state matrix of the tth processing layer can integrate feature information from different sources, better reflect the state of the current processing layer, and provide an accurate basis for the generation of subsequent output matrices and the processing of the entire target detection model.

[0126] The output matrix is ​​generated according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix, and the state matrix of the t-th processing layer can be combined with the first matrix obtained by linear processing to generate an output matrix that integrates various features as the final result of the t-th processing layer. Among them, the zero-order hold result of the state matrix of the t-th processing layer can be multiplied by the first matrix to obtain the output matrix.

[0127] The output matrix of the t-1th processing layer can be used as the input matrix of the tth processing layer. This processing method transmits information layer by layer, and correlates the output and input of different processing layers, so that the output matrix of each processing layer integrates the feature information accumulated by the previous processing layers, so that the target detection model can gradually extract more complex and representative target features, and improve the accuracy and robustness of target detection.

[0128] The implementation process of the second linear processing module 421 included in the above-mentioned second branch 402 can be similar to that of the first linear processing module 411, and the implementation process of the second activation module 422 included in the above-mentioned second branch 402 can be similar to that of the first activation module, so they are not repeated here and can be referred to each other.

[0129] The fusion module 403 fuses the outputs of the first branch and the second branch. The specific fusion method may be addition or multiplication.

[0130] It should be noted that the first linear processing module and the second linear processing module can perform linear processing on the input to increase the dimension. The third linear processing module 404 can perform linear processing on the input (fusion result, etc.) to restore the dimension.

[0131] The first linear processing module and the second linear processing module increase the dimension, which can allow the data to be represented in a higher dimensional space, and help capture richer and more detailed feature information. The data after the dimension is increased by the first two modules is restored by the third linear processing module 404. This process can integrate and optimize the rich features extracted and represented in the high-dimensional space, present them in a more appropriate and refined form, remove some unnecessary redundant information, so that the final output result can contain the features required for accurate target identification, and will not affect subsequent processing and judgment due to too much complex information, thereby improving the accuracy and efficiency of the entire processing flow in tasks such as target detection.

[0132] Moreover, the fusion of the first branch and the second branch can integrate the advantages of both, enrich the types of features, enable the target model to identify the target object from multiple angles, and enhance the accuracy of target detection.

[0133] The spatial pyramid pooling module is used to perform spatial pyramid pooling processing on the input features. The spatial pyramid pooling processing may include: convolution operation and maximum pooling operation, which can achieve deep fusion of input features.

[0134] Reference Figure 5 , shows a schematic diagram of the structure of a spatial pyramid pooling module according to an embodiment of the present application, wherein the spatial pyramid pooling module specifically includes: a thirteenth convolution module 501, p maximum pooling modules 502, a fifth splicing module 503, and a fourteenth convolution module 504. p may be a positive integer greater than 1, and the value of p in the figure is 3.

[0135] Among them, the thirteenth convolution module 501 is used to perform convolution processing on the input features to obtain the convolution result A.

[0136] The p maximum pooling modules 502 are respectively used to perform maximum pooling processing on the input features to obtain corresponding p maximum pooling processing results.

[0137] The convolution result A and the p maximum pooling processing results are respectively input into the fifth splicing module 503, and the fifth splicing module 603 performs deep fusion on the convolution result A and the p maximum pooling processing results to obtain a corresponding deep fusion result.

[0138] The fourteenth convolution module 504 is used to perform convolution processing on the depth fusion result to obtain a convolution result B. The convolution result B is provided to the neck network.

[0139] The process of updating the first weight matrix and the second weight matrix according to the input matrix in the embodiment of the present application specifically includes: determining the statistics corresponding to the dimensional features in the input matrix; determining the target dimensional features for which the statistics meet preset conditions; and increasing the corresponding weight values ​​for the rows or columns corresponding to the target dimensional features in the weight matrix and the second weight matrix.

[0140] The update of the first weight matrix and the second weight matrix can make the weight matrix of the target detection model adapt to the feature changes of the input data. The weight matrix is ​​adjusted according to the statistics of the dimensional features of the input matrix, so that the target detection model can focus on the valuable target dimensional features in the input data and improve the effectiveness of feature extraction of the input data.

[0141] The weight values ​​corresponding to the target dimension features are increased in a targeted manner, so that the target detection model pays more attention to these target dimension features in the subsequent processing process, thereby improving the accuracy and reliability of the target detection results.

[0142] In a specific implementation, the statistic specifically includes: variance or covariance, and the preset condition specifically includes: the statistic is greater than a threshold.

[0143] Reference Figure 6 , shows a schematic diagram of the structure of a neck network of an embodiment of the present application, the neck network specifically includes: a first-level feature fusion module, a second-level feature fusion module and a third-level feature fusion module;

[0144] The first-level feature fusion module includes: a sixth convolution module 601, a first upsampling module 602, a first splicing module 603, a second long sequence processing module A604, a seventh convolution module 605, a second upsampling module 606, a second splicing module 607, a second long sequence processing module B608 and a tenth convolution module 615 connected in series in sequence;

[0145] The second-level feature fusion module includes: an eighth convolution module 609, a third concatenation module 610, a second long sequence processing module C611, and an eleventh convolution module 616 connected in series in sequence;

[0146] The third-level feature fusion module includes: a ninth convolution module 612, a fourth concatenation module 613, a second long sequence processing module D614 and a twelfth convolution module 617 connected in series.

[0147] The sixth convolution module 601 receives the first image feature C, and performs convolution processing on the first image feature C to obtain a sixth convolution result.

[0148] The first upsampling module 602 is used to perform upsampling processing on the sixth convolution result to obtain a first upsampling processing result. In target detection, upsampling processing can be used to improve the resolution of image features. For example, a low-resolution feature map can be restored to a higher resolution to make the feature information richer, which helps to more accurately locate and identify the target object.

[0149] The second long sequence processing module A604 may have the same structure as the first long sequence processing module, and the structure of the second long sequence processing module A604 is not described in detail herein.

[0150] Specifically, the second long sequence processing module A604 includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result;

[0151] The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using the first weight matrix, linearly processing the input matrix to obtain the first matrix; using the zero-order hold technology, processing the first matrix to obtain the second matrix; updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the t-th processing layer; obtaining the output matrix of the t-th processing layer according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t can be a positive integer;

[0152] The second long sequence processing module A604 processes the first up-sampling processing result to obtain a third processing result.

[0153] The second stitching module 607 stitches the third processing result and the first image feature B to obtain a first stitching result. The above stitching process can increase the number of channels or the number of feature dimensions of the image feature.

[0154] The second long sequence processing module A604 processes the first splicing result to obtain a fourth processing result.

[0155] The seventh convolution module 605 performs convolution processing on the fourth processing result to obtain a seventh convolution result.

[0156] The second up-sampling module 606 performs up-sampling processing on the seventh convolution result to obtain a second up-sampling processing result.

[0157] The second stitching module 607 stitches the second upsampling result and the first image feature A to obtain a second stitching result.

[0158] The second long sequence processing module B608 processes the first splicing result to obtain a fifth processing result.

[0159] The tenth convolution module 615 performs convolution processing on the fifth processing result to obtain a second image feature A, and the second image feature A is output to the detection network.

[0160] The eighth convolution module 609 performs convolution processing on the fifth processing result to obtain an eighth convolution result.

[0161] The third splicing module 610 splices the eighth convolution result and the seventh convolution result to obtain a third splicing result.

[0162] The second long sequence processing module C611 processes the third splicing result to obtain a sixth processing result.

[0163] The eleventh convolution module 616 performs convolution processing on the sixth processing result to obtain a second image feature B, and the second image feature B is output to the detection network.

[0164] The ninth convolution module 612 performs convolution processing on the sixth processing result to obtain a ninth convolution result.

[0165] The fourth splicing module 613 splices the ninth convolution result and the sixth convolution result to obtain a fourth splicing result.

[0166] The second long sequence processing module D614 processes the fourth splicing result to obtain a seventh processing result.

[0167] The twelfth convolution module 617 performs convolution processing on the seventh processing result to obtain a second image feature C, and the second image feature C is output to the detection network.

[0168] In the first branch included in the second long sequence processing module, the linear processing module, the convolution processing module, the first activation module and the selective processing module work together. Among them, the linear processing module performs a preliminary linear transformation on the input, laying the foundation for subsequent processing; the convolution processing module can extract local features of the image and mine feature information at different positions; the first activation module introduces nonlinear factors to enhance the expression ability of the target detection model. The linear processing module and the second activation module in the second branch also play a role in feature processing, and the fusion module fuses the outputs of the two branches, so that the image features can be processed and integrated from multiple dimensions, avoiding the information loss or one-sidedness that may be caused by a single processing method, and comprehensively and deeply mining the image feature information.

[0169] In the selective processing module, the update mechanism of the state matrix is ​​of great significance. The first matrix is ​​obtained by linearly processing the input matrix using the first weight matrix. This step is based on the remapping of the input data based on the weights. The second matrix is ​​obtained by processing the first matrix using the zero-order hold technology. This technology maintains key features within a specific position interval and helps to retain information that is important for long-distance dependency processing in target detection. The state matrix is ​​updated according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix. This dynamic update mechanism enables the weights to be adaptively adjusted according to changes in the input matrix. By focusing on valuable target dimension features, the effectiveness and accuracy of input data feature extraction are significantly improved, which enables the target detection model to better adapt to feature changes in different input data.

[0170] In addition, in the overall process of the neck network, the role of the second long sequence processing module runs through multiple links. When processing the upsampling results, it optimizes the new feature information generated by the upsampling operation to improve the resolution, so that this information is more in line with the needs of target detection. For different splicing results, whether it is the splicing of the upsampling results with other image features or the splicing between the convolution results, the second long sequence processing module can further optimize the features. In this way, the information contained in the second image features A, B, and C output later is richer and more accurate. This helps the detection network to more accurately locate and identify target objects when performing target detection, reduce misjudgments and missed judgments, and thus significantly improve the overall performance of target detection. At the same time, the splicing process increases the number of channels or feature dimensions of the image features, and the second long sequence processing module effectively integrates and optimizes these newly added dimensions and channel features, enhancing the network's ability to process complex image features, enabling it to cope with more complex target detection scenarios.

[0171] In the embodiment of the present application, the detection network may include: at least one detection module. The detection module may be used to perform classification and regression calculations based on a plurality of second image features to obtain a target detection result, and the target detection result may specifically include: a category label and a bounding box of a target object in the image to be detected.

[0172] The embodiment of the present application can output the target detection result. Alternatively, the embodiment of the present application can issue corresponding prompt information for the target object. For example, in a security monitoring system, when a target object with abnormal behavior is detected, an alarm can be issued to the user in time so that the user can take corresponding measures. The prompt information can be in various forms such as sound alarm, image flashing, SMS notification, etc., which can be selected according to specific application requirements.

[0173] In an embodiment of the present application, the training process of the target detection model may include: forward propagation and back propagation.

[0174] Among them, forward propagation can calculate the prediction information of the target detection result in sequence from the backbone network to the detection network according to the parameters of the target detection model. The prediction information is used to determine the loss information.

[0175] Backward Propagation can calculate and update the parameters of the target detection model in sequence according to the loss information, in the order from the detection network to the backbone network. The target detection model usually adopts the structure of a neural network, and the parameters of the target detection model may include: parameters such as the weight matrix of the neural network. Among them, during the back propagation process, the gradient information of the parameters of the target detection model can be determined, and the parameters of the target detection model can be updated using the gradient information. For example, back propagation can calculate and store the gradient information of the parameters of the target detection model in sequence from the detection network to the neck network and then to the backbone network according to the chain rule in calculus.

[0176] In one embodiment, the training process of the target detection model includes:

[0177] Step B1, inputting an image sample into a target detection model, and the target detection model outputs a prediction result corresponding to the image sample; the prediction result includes: prediction box information corresponding to the target object category; and real box information corresponding to the image sample;

[0178] Step B2, determining loss information corresponding to the real box information and the predicted box information; the loss information includes: cross entropy loss corresponding to the predicted box category, intersection-over-union loss corresponding to the predicted box position, and focus loss corresponding to the predicted box position; the predicted box represents a predicted bounding box;

[0179] Step B3: Update the parameters of the target detection model according to the loss information.

[0180] After the training of the target detection model is completed, the trained target detection model can be used to detect the image to be detected to obtain the target detection result corresponding to the image to be detected.

[0181] In summary, the target detection method of the embodiment of the present application sets a long sequence processing model in the backbone network of the target detection model. The selective processing module of the long sequence processing model includes: multiple processing layers. And, according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix, the state matrix of the t-1th processing layer is updated to obtain the state matrix of the tth processing layer.

[0182] The update mechanism of the above-mentioned state matrix can effectively integrate past information and current information, so that the target detection model can remember the past information that appeared earlier in the image features, and can use this past information to establish long-distance dependencies in subsequent processing. Therefore, the update mechanism of the above-mentioned state matrix can effectively improve the ability of the target detection model to handle long-distance dependencies, so that the target detection model can more accurately identify the target object when facing complex images, and thus can effectively reduce the probability of errors in the bounding box in the target detection results. For example, in an image containing multiple target objects that are mutually occluded or far away, the target detection model can use the established long-distance dependencies to accurately outline the complete outline and position of each target object, reduce false detections and missed detections, and thereby improve the accuracy and robustness of target detection.

[0183] In addition, the convolution module in the backbone network can extract local image features of the image to be detected, so that the first long sequence processing module can mine the long-distance dependencies based on these local image features. The spatial pyramid pooling module in the backbone network can fuse the output image features of the first long sequence processing module with other image features at different scales. In this way, the fused image features contain both long-distance dependency information and multi-scale characteristics. Therefore, the expressiveness and robustness of image features can be improved, so that the target detection model can better cope with target detection at different scales and complex scenes, and enhance the target detection model's recognition ability and positioning accuracy of target objects in images.

[0184] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0185] Based on the above embodiment, this embodiment further provides a target detection device, referring to Figure 7 As shown in the structural schematic diagram, the device specifically includes: a receiving module 701 and a target detection module 702.

[0186] The receiving module 701 is used to receive the image to be detected;

[0187] The target detection module 702 is used to perform target detection on the image to be detected by using a target detection model to obtain a corresponding target detection result;

[0188] The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module;

[0189] The first long sequence processing module includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result;

[0190] The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using a first weight matrix, linearly processing the input matrix to obtain a first matrix; using a zero-order hold technique, processing the first matrix to obtain a second matrix; updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the t-th processing layer; obtaining the output matrix of the t-th processing layer according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t is a positive integer;

[0191] The target detection module 702 includes:

[0192] A feature extraction module 721 is used to determine a plurality of first image features corresponding to the image to be detected by using the backbone network;

[0193] A feature fusion module 722 is used to use the neck network to perform feature fusion processing according to the multiple first image features to obtain multiple second image features;

[0194] The result determination module 723 is used to determine the target detection result according to the multiple second image features using the detection network.

[0195] Optionally, updating the first weight matrix and the second weight matrix according to the input matrix includes:

[0196] Determine the statistics corresponding to the dimensional features in the input matrix;

[0197] Determine the target dimension characteristics that the statistics meet the preset conditions;

[0198] For the rows or columns corresponding to the target dimensional features in the weight matrix and the second weight matrix, the corresponding weight values ​​are increased.

[0199] Optionally, the statistic includes: variance or covariance, and the preset condition includes: the statistic is greater than a threshold.

[0200] Optionally, updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix includes:

[0201] Multiply the second matrix and the input matrix to obtain a first product result;

[0202] Multiplying the zero-order hold result of the second weight matrix by the state matrix of the t-1th processing layer to obtain a second product result;

[0203] The first product result and the second product result are added to obtain the state matrix of the tth processing layer.

[0204] Optionally, the backbone network includes: a first convolution module, a second convolution module, a first long sequence processing module A, a third convolution module, a first long sequence processing module B, a fourth convolution module, a first long sequence processing module C, a fifth convolution module, a first long sequence processing module D and a spatial pyramid pooling module connected in sequence;

[0205] Among them, the first long sequence processing module B outputs the first image feature A to the neck network; the first long sequence processing module C outputs the first image feature B to the neck network; and the spatial pyramid pooling module outputs the first image feature C to the neck network.

[0206] Optionally, the neck network includes: a first-level feature fusion module, a second-level feature fusion module and a third-level feature fusion module;

[0207] The first-level feature fusion module includes: a sixth convolution module, a first upsampling module, a first splicing module, a second long sequence processing module A, a seventh convolution module, a second upsampling module, a second splicing module, a second long sequence processing module B and a tenth convolution module connected in series in sequence;

[0208] The second-level feature fusion module includes: an eighth convolution module, a third splicing module, a second long sequence processing module C, and an eleventh convolution module connected in series in sequence;

[0209] The third-level feature fusion module includes: a ninth convolution module, a fourth splicing module, a second long sequence processing module D and a twelfth convolution module connected in series in sequence.

[0210] Optionally, the spatial pyramid pooling module includes: a thirteenth convolution module, p maximum pooling modules, a fifth splicing module and a fourteenth convolution module; p is a positive integer greater than 1.

[0211] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0212] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0213] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Figure 8 An exemplary apparatus 1100 that can be used to implement various embodiments described in this application is schematically shown.

[0214] For one embodiment, Figure 8An exemplary apparatus 1100 is shown having one or more processors 1102, a control module (chip set) 1104 coupled to at least one of the (one or more) processors 1102, a memory 1106 coupled to the control module 1104, a non-volatile memory / storage device 1108 coupled to the control module 1104, one or more input / output devices 1110 coupled to the control module 1104, and a network interface 1112 coupled to the control module 1104.

[0215] The processor 1102 may include one or more single-core or multi-core processors, and the processor 1102 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1100 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0216] In some embodiments, the device 1100 may include one or more computer-readable media (e.g., memory 1106 or non-volatile memory / storage device 1108) having instructions 1114 and one or more processors 1102 configured to execute the instructions 1114 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0217] For one embodiment, the control module 1104 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1102 and / or any suitable device or component in communication with the control module 1104 .

[0218] The control module 1104 may include a memory controller module to provide an interface to the memory 1106. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0219] The memory 1106 may be used, for example, to load and store data and / or instructions 1114 for the device 1100. For one embodiment, the memory 1106 may include any suitable volatile memory, such as a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1106 may include a double data rate type four synchronous dynamic random access memory.

[0220] For one embodiment, the control module 1104 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1108 and the input / output device(s) 1110 .

[0221] For example, non-volatile memory / storage 1108 may be used to store data and / or instructions 1114. Non-volatile memory / storage 1108 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0222] The non-volatile memory / storage device 1108 may include storage resources that are physically part of the device on which the apparatus 1100 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1108 may be accessed via (one or more) input / output devices 1110 over a network.

[0223] The input / output device(s) 1110 may provide an interface for the apparatus 1100 to communicate with any other appropriate device, and the input / output device 1110 may include a communication component, an audio component, a sensor component, etc. The network interface 1112 may provide an interface for the apparatus 1100 to communicate through one or more networks, and the apparatus 1100 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.

[0224] For one embodiment, at least one of the processor(s) 1102 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be packaged together with the logic of one or more controllers of the control module 1104 to form a system-in-package. For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with the logic of one or more controllers of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with the logic of one or more controllers of the control module 1104 to form a system-on-chip.

[0225] In various embodiments, the device 1100 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1100 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1100 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.

[0226] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0227] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0228] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0229] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0230] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0232] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0233] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0234] The target detection method and device, an electronic device and a machine-readable medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application sets. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Receiving an image to be detected; Using the target detection model, performing target detection on the image to be detected to obtain a corresponding target detection result; The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module; The first long sequence processing module includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result; The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using a first weight matrix, linearly processing the input matrix to obtain a first matrix; using a zero-order hold technique, processing the first matrix to obtain a second matrix; updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the t-th processing layer; obtaining the output matrix of the t-th processing layer according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t is a positive integer; The method of using the target detection model to perform target detection on the image to be detected includes: The backbone network determines a plurality of first image features corresponding to the image to be detected; The neck network performs feature fusion processing according to the multiple first image features to obtain multiple second image features; The detection network determines the target detection result according to the multiple second image features.

2. The method according to claim 1, characterized in that The updating the first weight matrix and the second weight matrix according to the input matrix comprises: Determine the statistics corresponding to the dimensional features in the input matrix; Determine the target dimension characteristics that the statistics meet the preset conditions; For the rows or columns corresponding to the target dimensional features in the weight matrix and the second weight matrix, the corresponding weight values ​​are increased.

3. The method according to claim 2, characterized in that The statistic includes: variance or covariance, and the preset condition includes: the statistic is greater than a threshold.

4. The method according to claim 1, characterized in that: The updating of the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix comprises: Multiply the second matrix and the input matrix to obtain a first product result; Multiplying the zero-order hold result of the second weight matrix by the state matrix of the t-1th processing layer to obtain a second product result; The first product result and the second product result are added to obtain the state matrix of the tth processing layer.

5. The method according to claim 1, characterized in that The backbone network includes: a first convolution module, a second convolution module, a first long sequence processing module A, a third convolution module, a first long sequence processing module B, a fourth convolution module, a first long sequence processing module C, a fifth convolution module, a first long sequence processing module D and a spatial pyramid pooling module connected in sequence; Among them, the first long sequence processing module B outputs the first image feature A to the neck network; the first long sequence processing module C outputs the first image feature B to the neck network; and the spatial pyramid pooling module outputs the first image feature C to the neck network.

6. The method according to claim 1, characterized in that The neck network includes: a first-level feature fusion module, a second-level feature fusion module and a third-level feature fusion module; The first-level feature fusion module includes: a sixth convolution module, a first upsampling module, a first splicing module, a second long sequence processing module A, a seventh convolution module, a second upsampling module, a second splicing module, a second long sequence processing module B and a tenth convolution module connected in series in sequence; The second-level feature fusion module includes: an eighth convolution module, a third splicing module, a second long sequence processing module C, and an eleventh convolution module connected in series in sequence; The third-level feature fusion module includes: a ninth convolution module, a fourth splicing module, a second long sequence processing module D and a twelfth convolution module connected in series in sequence.

7. The method according to any one of claims 1 to 6, characterized in that: The spatial pyramid pooling module includes: a thirteenth convolution module, p maximum pooling modules, a fifth splicing module and a fourteenth convolution module; p is a positive integer greater than 1.

8. A target detection device, characterized in that: The device comprises: A receiving module, used for receiving an image to be detected; The target detection module is used to perform target detection on the image to be detected by using the target detection model to obtain the corresponding target detection result; The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: at least one convolution module, at least one first long sequence processing module and a spatial pyramid pooling module; The first long sequence processing module includes: a first branch, a second branch, a fusion module and a third linear processing module; the first branch includes: a first linear processing module, a convolution processing module, a first activation module, and a selective processing module; the second branch includes: a second linear processing module and a second activation module; the fusion module fuses the outputs of the first branch and the second branch to obtain a fusion result; the third linear processing module performs linear processing on the fusion result; The selective processing module includes a plurality of processing layers, each processing layer corresponds to a state matrix, and the processing process of the t-th processing layer includes: using a first weight matrix, linearly processing the input matrix to obtain a first matrix; using a zero-order hold technique, processing the first matrix to obtain a second matrix; updating the state matrix of the t-1th processing layer according to the zero-order hold result of the second weight matrix, the second matrix and the input matrix to obtain the state matrix of the t-th processing layer; obtaining the output matrix of the t-th processing layer according to the zero-order hold result of the state matrix of the t-th processing layer and the first matrix; wherein the first weight matrix and the second weight matrix are updated according to the input matrix; t is a positive integer; The target detection module comprises: A feature extraction module, used to determine a plurality of first image features corresponding to the image to be detected by using the backbone network; A feature fusion module, used to utilize the neck network to perform feature fusion processing according to the plurality of first image features to obtain a plurality of second image features; The result determination module is used to determine the target detection result according to the plurality of second image features by using the detection network.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 7.

10. A machine-readable medium having executable codes stored thereon, which, when executed, cause a processor to perform the method according to any one of claims 1 to 7.

Images