Target detection and classification methods, devices, equipment, and media based on embedded devices

Abstract

This disclosure presents a method, apparatus, device, and medium for target detection and classification based on embedded devices. One specific implementation of the method includes: receiving an initial location information set; determining an initial size value set; determining a first threshold size value and a second threshold size value; generating a target size value set; filtering a subset of the initial location information as the target location information set; inputting the target location information set and the image to be detected into a feature extraction layer to obtain a first target feature space set; inputting the first target feature space set into a feature space scaling layer to obtain a second target feature space set; and inputting the second target feature space set into a fully connected layer to obtain a classification result set. This implementation optimizes the target detection and classification process on embedded devices by constraining the feature space set input to the feature space scaling layer, thereby improving the processing efficiency and stability of target detection and image classification.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to target detection and classification methods, apparatus, devices, and media based on embedded devices. Background Technology

[0002] Object detection and classification mainly involves two steps: determining the location of objects and predicting their categories. Because the object detection and classification process on the central processing unit (CPU) and neural network processor requires repeated image cropping and scaling, the processing efficiency is low. Therefore, optimizing the object detection and classification process between the CPU and neural network processor is crucial. A typical approach for object detection and classification methods is as follows: First, a subset of images corresponding to the detection result set is extracted from the image to be detected. Then, this subset is cropped and scaled to obtain a processed image set. Finally, the processed image set is sent to the neural network processor for classification processing to obtain the classification result set.

[0003] However, the inventors discovered that when using the above method for target detection and classification, the following technical problems often arise:

[0004] First, because the images to be detected are cropped, the sub-image sets are clipped and scaled, and the processed sub-images are repeatedly sent to the neural network processor for classification, the sub-images occupy a large amount of bandwidth and central processing unit memory, resulting in low processing efficiency and stability of the target detection model and classification neural network.

[0005] Second, because the non-maximum suppression algorithm selects the candidate box with the highest classification confidence as the optimal candidate box, but the classification confidence and location information are not strongly correlated, the localization accuracy is low, resulting in a high false negative rate.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure propose target detection and classification methods, apparatuses, devices, and media based on embedded devices to solve one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide a target detection and classification method based on an embedded device, comprising: receiving an initial position information set corresponding to a set of target objects in an image to be detected sent by a central processing unit; determining an initial size value set for the set of target objects based on the initial position information set; determining a first threshold size value and a second threshold size value corresponding to the input feature space of a feature space scaling layer included in a classification neural network, wherein the classification neural network further includes a feature extraction layer and a fully connected layer; generating a target size value set based on the first threshold size value, the second threshold size value, and the initial size value set; filtering out a subset of initial position information corresponding to the target size value set from the initial position information set as a target position information set; inputting the target position information set and the image to be detected into the feature extraction layer to obtain a first target feature space set; inputting the first target feature space set into the feature space scaling layer to obtain a second target feature space set with a preset size value, wherein the feature space scaling layer is a neural network layer that performs constraint processing on the input first target feature space set to obtain the second target feature space set with the preset size value; and inputting the second target feature space set into the fully connected layer to obtain a classification result set.

[0010] Secondly, some embodiments of this disclosure provide a target detection and classification apparatus based on an embedded device, comprising: a receiving unit configured to receive an initial position information set corresponding to a set of target objects in an image to be detected, sent by a central processing unit; a first determining unit configured to determine an initial size value set for the set of target objects based on the initial position information set; a second determining unit configured to determine a first threshold size value and a second threshold size value corresponding to the input feature space of a feature space scaling layer included in a classification neural network, wherein the classification neural network further includes: a feature extraction layer and a fully connected layer; a generating unit configured to generate a target size value set based on the first threshold size value, the second threshold size value, and the initial size value set; and a filtering unit. The first input unit is configured to filter out a subset of initial position information corresponding to the target size value set from the initial position information set, as the target position information set; the second input unit is configured to input the target position information set and the image to be detected into the feature extraction layer to obtain a first target feature space set; the third input unit is configured to input the first target feature space set into the feature space scaling layer to obtain a second target feature space set with a preset size value, wherein the feature space scaling layer is a neural network layer that constrains the input first target feature space set to obtain the second target feature space set with a preset size value; the fourth input unit is configured to input the second target feature space set into the fully connected layer to obtain a classification result set.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0013] The above embodiments of this disclosure have the following beneficial effects: The target detection and classification method based on embedded devices in some embodiments of this disclosure optimizes the target detection and classification process on embedded devices by constraining the feature space set input to the feature space size scaling layer, thereby improving the processing efficiency and stability of target detection and image classification. Specifically, the reason for the low processing efficiency and stability of related target detection models and classification neural networks is that the cropping and scaling of the sub-image set, and the repeated sending of the processed sub-images to the neural network processor for classification, cause the sub-images to occupy a large amount of transmission bandwidth and central processing unit memory space, thus resulting in low processing efficiency and stability of the target detection model and classification neural network. Based on this, the target detection and classification method based on embedded devices in some embodiments of this disclosure can first receive the initial position information set corresponding to the target object set in the image to be detected sent by the central processing unit. Based on the initial position information set, an initial size value set for the target object set is determined. Here, the obtained initial position information set and initial size value set facilitate the subsequent determination of the first threshold size value and the second threshold size value of the feature space size scaling layer. Secondly, the first threshold size and the second threshold size corresponding to the input feature space of the feature space scaling layer in the classification neural network are determined. The classification neural network also includes a feature extraction layer and a fully connected layer. Determining the first threshold size avoids inputting sub-image sets smaller than the first threshold size to the feature extraction layer, which could cause errors in the classification neural network. Determining the second threshold size facilitates the subsequent determination of the target size set. Next, a target size set is generated based on the first threshold size, the second threshold size, and the initial size set. This generated target size set avoids errors caused by sub-image sets smaller than the first threshold size. For sub-image sets larger than the second threshold size, it facilitates determining the method for generating the target size set and facilitates subsequent classification processing. Then, a subset of initial position information corresponding to the target size set is selected from the initial position information set as the target position information set. Determining the target position information set facilitates the subsequent determination of the sub-image set corresponding to the target position set and the image to be detected. Finally, the target position information set and the image to be detected are input to the feature extraction layer to obtain the first target feature space set. Here, by directly reading the sub-image set corresponding to the target location information set from the image to be detected, the storage and processing of the captured image by the central processing unit can be eliminated, reducing the load on the central processing unit and memory space, and improving the processing efficiency and stability of image detection and classification of embedded devices.Then, the first target feature space set is input to the feature space scaling layer to obtain a second target feature space set with a preset size. The feature space scaling layer is a neural network layer that constrains the input first target feature space set to obtain the second target feature space set with the preset size. Here, the feature space scaling layer can accept target feature spaces of different sizes, thereby eliminating the need for image cropping and scaling by the central processing unit (CPU), improving the efficiency and stability of target detection and classification, and reducing the transmission bandwidth between the CPU and the neural network processor. Finally, the second target feature space set is input to the fully connected layer to obtain the classification result set. Therefore, this target detection and classification method based on embedded devices optimizes the target detection and classification process on embedded devices by constraining the feature space set input to the feature space scaling layer, thereby improving the efficiency and stability of target detection and image classification. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 This is a flowchart of some embodiments of the target detection and classification method based on embedded devices according to the present disclosure;

[0016] Figure 2 This is a schematic diagram of the structure of some embodiments of the target detection and classification device based on embedded devices according to the present disclosure;

[0017] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Figure 1 A flow 100 of some embodiments of an embedded device-based target detection and classification method according to the present disclosure is shown. The embedded device-based target detection and classification method includes the following steps:

[0025] Step 101: Receive the initial position information set corresponding to the target object set in the image to be detected sent by the central processing unit.

[0026] In some embodiments, the execution entity (e.g., a neural network processor) of the above-described target detection and classification method based on an embedded device can receive an initial position information set corresponding to the target object set in the image to be detected, sent by a central processing unit (CPU), via a wired or wireless connection. The CPU can be a processor that transmits data with the execution entity. Both the CPU and the execution entity are processors located on the embedded device. The image to be detected can be an image that needs to be detected and classified. The target objects in the target object set can be objects included in the image to be detected. The initial position information set can represent the position information of the target object set. The initial position information in the initial position information set can be represented as (x_min, y_min, x_max, y_max). (x_min, y_min) represents the position information of the upper left corner of the target object. x_min represents the x-coordinate of the upper left corner of the target object, and y_min represents the y-coordinate of the upper left corner of the target object. (x_max, y_max) represents the position information of the lower right corner of the target object. x_max represents the x-coordinate of the lower right corner of the target object, and y_max represents the y-coordinate of the lower right corner of the target object. For example, the image to be detected may be an image of a target object including at least one of the following: a cat, a dog, a person, and a bicycle. The target object set may include: cats, dogs, people, and bicycles. The initial position information set may include: initial position information for a cat (10, 135, 54, 165), initial position information for a dog (20, 153, 70, 191), initial position information for a person (87, 131, 127, 183), and initial position information for a bicycle (35, 91, 105, 121).

[0027] Optionally, before receiving the initial position information set corresponding to the target object set in the image to be detected sent by the central processing unit, the following steps may be included:

[0028] The first step is to receive the image to be detected sent by the central processing unit.

[0029] The second step involves inputting the image to be detected into the object detection model to obtain a detection result set. This object detection model can be a model that performs object detection on the input image. For example, the object detection model can be at least one of the following: R-CNN (Region with CNN features) model and Fast R-CNN (Fast Region-based Convolutional Network) model. The detection results in the result set can be representations of the target object's location information and confidence level, expressed as tensor matrices. The confidence level can represent the probability values ​​of different candidate boxes containing the same target object.

[0030] The third step is to send the detection result set to the central processing unit. It should be noted that the central processing unit decodes and filters the detection result set to obtain the initial location information set.

[0031] In some optional implementations of certain embodiments, the aforementioned initial location information set may be obtained through the following steps:

[0032] The first step is to decode the above detection result set to obtain a decoded detection result set. This decoded detection result set can be a set of candidate bounding boxes for the target objects in the image to be detected. Each target object can have multiple candidate bounding boxes. The candidate bounding boxes in the above set represent the location information of the target objects and the classification confidence level of the selected target objects. Each candidate bounding box in the above set includes: the location information of the candidate bounding box and the confidence value of the target objects included in the candidate bounding box.

[0033] The second step involves filtering candidate boxes on the decoded detection result set to obtain an initial location information set. The initial location information in this set represents the location of the target object. In practice, the executing entity can utilize the Non-Maximum Suppression (NMS) algorithm and a preset intersection-union (IU) threshold to filter candidate boxes on the decoded detection result set to obtain the initial location information set. The preset IU threshold represents the degree of overlap between any two candidate boxes. The preset overlap value can be 0.5.

[0034] In some optional implementations of certain embodiments, the above-described candidate box filtering process for the decoded detection result set to obtain the initial position information set may include the following steps:

[0035] The first step involves clustering the decoded detection result set to obtain a clustered decoded detection result set for each target object in the image to be detected. Each clustered decoded detection result set represents at least one decoded detection result for a single target object. In practice, the execution entity can utilize DBSCAN (Density-Based Spatial Clustering of Applications with Noise) to cluster the decoded detection result set, obtaining a clustered decoded detection result set for each target object in the image to be detected.

[0036] The second step involves performing the following determination steps for each of the multiple clustering decoding detection result sets mentioned above:

[0037] Sub-step 1: Select the clustering decoding result with the highest confidence value from the above clustering decoding result set as the target clustering decoding result.

[0038] Sub-step 2 involves standardizing the location information corresponding to each cluster-decoded detection result in the cluster-decoded detection result set to generate standardized location information, thus obtaining a standardized location information set. This standardized location information can be location information whose coordinate values ​​have been converted to coordinate values ​​between 0 and 1.

[0039] As an example, the aforementioned execution entity can first select the maximum, minimum, maximum, and minimum values ​​of the horizontal coordinates, and the vertical coordinates, from the clustering decoding detection result set. Then, for each clustering decoding detection result in the clustering decoding detection result set, the following determination steps are performed: The difference between each horizontal coordinate in the clustering decoding detection result and the aforementioned minimum horizontal coordinate is determined as the first cluster horizontal coordinate set. The difference between the maximum and minimum horizontal coordinates is determined as the second cluster horizontal coordinate. The ratio of each first cluster horizontal coordinate in the first cluster horizontal coordinate set to the aforementioned second cluster horizontal coordinate is determined as the target cluster horizontal coordinate set. The difference between each vertical coordinate in the clustering decoding detection result and the aforementioned minimum vertical coordinate is determined as the first cluster vertical coordinate set. The difference between the maximum and minimum vertical coordinates is determined as the second cluster vertical coordinate. The ratio of each first cluster vertical coordinate in the first cluster vertical coordinate set to the aforementioned second cluster vertical coordinate is determined as the target cluster vertical coordinate set. The above target cluster abscissa and ordinate sets are defined as standardized location information. The detection result after the above clustering decoding is represented as (x1, y1, x2, y2, c). (x1, y1) represents the coordinates of the upper left corner of the candidate box. (x2, y2) represents the coordinates of the lower right corner of the candidate box. c represents the confidence score, i.e., the probability value of the classification result of the object selected by the candidate box.

[0040] Sub-step 3: Select the standardized location information corresponding to the detection results after the target clustering decoding from the above standardized location information set, and use it as the target standardized location information.

[0041] Sub-step 4: Based on the aforementioned standardized target location information and the aforementioned standardized location information set, generate a distance value set. The distance values ​​in this set represent the degree of similarity between each cluster-decoded detection result in the cluster-decoded detection result set and the target cluster-decoded detection result.

[0042] As an example, the aforementioned executing entity can perform the following determination steps for each standardized location information in the aforementioned standardized location information set: First, determine the absolute value of the difference between the first abscissa of the standardized location information and the first abscissa of the target standardized location information as the first abscissa value. Second, determine the absolute value of the difference between the first ordinate of the standardized location information and the first ordinate of the target standardized location information as the first ordinate value. Third, determine the first abscissa value and the first ordinate value as the first distance value. Next, determine the absolute value of the difference between the second abscissa of the standardized location information and the second abscissa of the target standardized location information as the second abscissa value. Subsequently, determine the absolute value of the difference between the second ordinate of the standardized location information and the second ordinate of the target standardized location information as the second ordinate value. Then, determine the second abscissa value and the second ordinate value as the second distance value. Finally, determine the first distance value and the second distance value as the distance value.

[0043] Sub-step 5: Select distance values ​​greater than or equal to a preset distance threshold from the above set of distance values ​​to obtain the target distance value set. The preset distance threshold can be any value within the range (0.3, 0.5).

[0044] Sub-step 6 involves determining the average deviation between each of the overlapping standardized location information in the overlapping standardized location information set corresponding to the target distance value set and the target standardized location information. The overlapping standardized location information set can be the standardized location information set in the clustering decoding detection result set corresponding to the target distance value set. In practice, the executing entity can first determine the absolute value of the difference between each standardized location information and the target standardized location information, obtaining a set of absolute values. Then, the sum of the absolute values ​​is determined as the target value. Finally, the average value of the target value is determined as the average deviation value.

[0045] Sub-step 7: The sum of the above-mentioned standardized target location information and the above-mentioned average deviation value is determined as the updated target decoding detection result.

[0046] The third step is to determine the location information corresponding to the multiple updated target decoding and detection results as the initial location information set.

[0047] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the second technical problem mentioned in the background: "Because the non-maximum suppression algorithm uses the candidate box with the highest classification confidence as the optimal candidate box, but the classification confidence and location information are not strongly correlated, the localization accuracy is low, resulting in a high false negative rate." Factors leading to a high false negative rate are often as follows: because the non-maximum suppression algorithm uses the candidate box with the highest classification confidence as the optimal candidate box, but the classification confidence and location information are not strongly correlated, the localization accuracy is low, resulting in a high false negative rate. Solving these factors can reduce the false negative rate. To achieve this effect, this disclosure first performs clustering processing on the above-described decoded detection result set to obtain a decoded detection result set corresponding to each target object in the above-described image to be detected, serving as multiple clustered decoded detection result sets. Here, multiple clustered decoded detection result sets facilitate subsequent determination of the target clustered decoded detection result set. Then, for each of the above clustering decoding detection result sets, the following determination steps are performed: The clustering decoding detection result with the highest confidence value is selected from the above clustering decoding detection result sets as the target clustering decoding detection result. The location information corresponding to each clustering decoding detection result in the above clustering decoding detection result sets is standardized to generate standardized location information, resulting in a standardized location information set. Here, standardization can better handle the proximity of clustering decoding detection results with different candidate box sizes, thereby improving the determination of the distance value between the target clustering decoding detection result and the clustering decoding detection result set. The standardized location information corresponding to the target clustering decoding detection result is selected from the above standardized location information set as the target standardized location information. Based on the above target standardized location information and the above standardized location information set, a distance value set is generated. Here, the distance value set fully utilizes the location information between the target clustering decoding detection result and the clustering decoding detection result set, which can suppress clustering decoding detection results with low positioning accuracy, thereby improving positioning accuracy. Distance values ​​greater than or equal to a preset distance threshold are selected from the aforementioned distance value set to obtain the target distance value set. This target distance value set facilitates subsequent updates to the target clustering decoding detection results. The average deviation value between each standardized location information in the corresponding overlapping standardized location information set and the aforementioned target standardized location information is determined. Here, the average deviation value characterizes the distance relationship between the target clustering decoding detection results and the clustering decoding detection result set. Using the average deviation value allows for better updates to the target clustering decoding detection results, thereby improving positioning accuracy and reducing false detection rate. The sum of the aforementioned target standardized location information and the aforementioned average deviation value is determined as the updated target decoding detection result.Here, updating the target clustering decoding detection results avoids selecting the candidate box with the highest confidence as the target clustering decoding detection result. Instead, it uses the distance between the target clustering decoding detection results and the target clustering decoding detection results to update the target clustering decoding detection results, resulting in more accurate positioning information, improving positioning accuracy and reducing the false positive rate. Finally, the location information corresponding to the multiple updated target decoding decoding detection results is determined as the initial location information set. Therefore, updating the location information corresponding to the target clustering decoding detection results using the average deviation value of the clustering decoding detection result set that is greater than or equal to a preset distance threshold can improve positioning accuracy and reduce the false positive rate.

[0048] Step 102: Determine the initial size value set for the target object set based on the initial position information set.

[0049] In some embodiments, the executing entity may determine an initial set of size values ​​for the target object set based on the initial position information set. The initial size values ​​in the initial size value set may represent the size of the target object. These initial size values ​​may include the length and width values ​​of the target object.

[0050] As an example, the aforementioned execution entity can first perform the following determination steps for each initial size value in the aforementioned initial size value set: First, determine the length value as the difference between the lower right and upper left horizontal coordinates corresponding to the initial position information. Second, determine the width value as the difference between the lower right and upper left vertical coordinates of the initial position information. Third, determine the length and width values ​​as the initial size values. Then, determine the resulting initial size value set as the initial size value set for the target object set.

[0051] Step 103: Determine the first threshold size value and the second threshold size value corresponding to the input feature space of the feature space size scaling layer included in the classification neural network.

[0052] In some embodiments, the execution entity may determine a first threshold size and a second threshold size corresponding to the input feature space of the feature space scaling layer included in the classification neural network. The classification neural network further includes a feature extraction layer and a fully connected layer. The classification neural network may be a neural network that classifies a set of target objects in an image to be detected. For example, the classification neural network may be a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network). The feature extraction layer may be a network layer that extracts features from the input image to be detected. The feature extraction layer may include multiple convolutional layers and multiple pooling layers. The feature space scaling layer may be a network layer that scales the size of the input feature space. For example, the feature space scaling layer may be a pooling layer. The feature space scaling layer may be a neural network layer that receives a feature space within a certain size range and performs constraint processing on the feature space within that size range to obtain a feature space of a preset size. The input feature space may represent the input data of the feature space scaling layer. The first threshold size may be the minimum size of the input feature space that the feature space scaling layer can receive. The second threshold size value mentioned above can be the maximum size value of the input feature space that the feature space size scaling layer can receive.

[0053] In some optional implementations of certain embodiments, determining the first threshold size value and the second threshold size value corresponding to the input feature space of the feature space size scaling layer of the classification neural network may include the following steps:

[0054] The first step is to determine the output size value corresponding to the output feature space of the aforementioned feature space scaling layer as the first threshold size value. Here, the aforementioned output feature space can represent the output result of the aforementioned feature space scaling layer. The aforementioned output size value can represent the size of the output feature space. The aforementioned output size value can be a fixed size value. In practice, the aforementioned execution entity can use an adaptive algorithm and the kernel size of the aforementioned feature space scaling layer to generate the output size value of the aforementioned feature space scaling layer.

[0055] The second step is to determine the size value as a second threshold size value, which is a preset integer multiple of the output size value. This preset integer multiple can be twice or more the output size value. It should be noted that the specific preset integer multiple needs to be determined based on the size value of the input image to be detected.

[0056] Step 104: Generate a target size value set based on the first threshold size value, the second threshold size value, and the initial size value set.

[0057] In some embodiments, the execution entity can generate a target size set based on the first threshold size value, the second threshold size value, and the initial size value set. The target size values ​​in the target size value set can be the size values ​​corresponding to the input feature space of the feature space scaling layer. It should be noted that sub-image sets corresponding to initial size values ​​smaller than the first threshold size value in the initial size value set will be deleted before being input to the feature space scaling layer. This reduces memory usage and avoids errors in the classification neural network. The sub-image set can be a set of images in the image to be detected whose positional information is smaller than the first threshold size value.

[0058] As an example, the aforementioned execution entity can filter out initial size values ​​that are greater than the first threshold size value and less than the second threshold size value from the aforementioned initial size value set, and use them as the target size value set.

[0059] In some optional implementations of certain embodiments, generating the target size value set based on the first threshold size value, the second threshold size value, and the initial size value set may include the following steps:

[0060] The first step is to determine, in response to the determination that there is an initial size value in the set of initial size values ​​that is greater than the second threshold size value, to define the initial size value that is greater than the second threshold size value as the first initial size value set.

[0061] The second step involves scaling each sub-image in the sub-image set to be detected corresponding to the first initial size value set, to generate scaled sub-images to be detected, thus obtaining a scaled sub-image set to be detected. The sub-image set to be detected is the sub-image set within the initial position information set corresponding to the first initial size value set in the image to be detected. The scaling process can be an integer multiple. For example, if the first initial size value is greater than twice the second threshold size value but less than three times the second threshold size value, then the sub-image to be detected corresponding to the first initial size value is scaled by half, meaning that feature extraction is performed every other pixel. For example, the first pixel of the sub-image to be detected is extracted, the second pixel is skipped, and the third pixel is extracted. If the first initial size value is greater than three times the second threshold size value but less than four times the second threshold size value, then the sub-image to be detected corresponding to the first initial size value is scaled by one-third, meaning that feature extraction is performed every two pixels. For example, the first pixel information of the sub-image to be detected is extracted, the second and third pixel information are skipped, and the fourth pixel information is extracted. It should be noted that the above-mentioned sub-image set to be detected is not a real image set. The sub-images to be detected in the above-mentioned sub-image set are sub-images directly read from the above-mentioned image to be detected, corresponding to the initial position information corresponding to the first initial size value. This can reduce the memory storage space of the central processing unit and improve the processing efficiency, performance and stability of object detection and classification.

[0062] The third step is to select initial size values ​​that are greater than the first threshold size value and less than the second threshold size value from the above initial size value set, and use them as the second initial size value set.

[0063] The fourth step is to determine the target size value set by combining the scaled size value set corresponding to the above-mentioned scaled sub-image set to be detected and the above-mentioned second initial size value set.

[0064] Step 105: Select a subset of initial position information that corresponds to the target size value set from the initial position information set, and use it as the target position information set.

[0065] In some embodiments, the execution entity can filter out a subset of initial position information corresponding to the target size value set from the initial position information set, as the target position information set. The target position information set can be the position information corresponding to the target size value set that the feature space size scaling layer can receive. There is a one-to-one correspondence between the target position information set and the target size value set. For example, the target position information set may include: first target position information and second target position information. The target size information set may include: first target size information and second target size information. The first target position information and the first target size information are the position and size information of the same target object. The second target position information and the second target size information are the position and size information of the same target object.

[0066] Step 106: Input the target location information set and the image to be detected into the feature extraction layer to obtain the first target feature space set.

[0067] In some embodiments, the execution entity can input the target location information set and the image to be detected into the feature extraction layer to obtain a first target feature space set. The first target feature space in the first target feature space set can be a feature space representing the semantic information of the image to be detected. It should be noted that by directly inputting the target location information set and the image to be detected into the feature extraction layer, the sub-image set corresponding to the target location information set can be directly read from the image to be detected, replacing the prior art of cropping the image to be detected to obtain a sub-image set. Then, the sub-image set is cropped and scaled. The effect achieved is that the CPU (Central Processing Unit) is avoided from cropping and processing the image to be detected in the prior art, and the CPU is also avoided from repeatedly submitting cropped and scaled images to the NPU (Neural-Network Processing Unit), thereby improving the stability and processing efficiency of target detection and classification, and reducing the transmission bandwidth of the CPU and NPU.

[0068] Step 107: Input the first target feature space set into the feature space size scaling layer to obtain the second target feature space set with a preset size value.

[0069] In some embodiments, the executing entity can input the first target feature space set into the feature space scaling layer to obtain a second target feature space set of a preset size. The feature space scaling layer is a neural network layer that constrains the input first target feature space set to obtain a second target feature space set of the preset size. The dimensions of each second target feature space in the second target feature space set are the same. The preset size can be a size determined based on the parameter matrix of the feature space scaling layer. The parameter matrix can be a matrix composed of network parameters of the feature space scaling layer. The feature space scaling layer can be a neural network layer that constrains the input first target feature space set and outputs a second target feature space set of the preset size. In practice, the executing entity can scale each first target feature space in the first target feature space set according to the preset kernel size of the feature space scaling layer to obtain a second target feature space set of the preset size. The preset kernel size can be a 3×3 pooling kernel.

[0070] Step 108: Input the second target feature space set into the fully connected layer to obtain the classification result set.

[0071] In some embodiments, the execution entity can input the second target feature space set into the fully connected layer to obtain a classification result set. The classification result in the classification result set can be the predicted classification result of the target object. For example, the target objects included in the image to be detected can be cats and dogs. Then the classification result can be cats and dogs.

[0072] The above embodiments of this disclosure have the following beneficial effects: The target detection and classification method based on embedded devices in some embodiments of this disclosure optimizes the target detection and classification process on embedded devices by constraining the feature space set input to the feature space size scaling layer, thereby improving the processing efficiency and stability of target detection and image classification. Specifically, the reason for the low processing efficiency and stability of related target detection models and classification neural networks is that the cropping and scaling of the sub-image set, and the repeated sending of the processed sub-images to the neural network processor for classification, cause the sub-images to occupy a large amount of transmission bandwidth and central processing unit memory space, thus resulting in low processing efficiency and stability of the target detection model and classification neural network. Based on this, the target detection and classification method based on embedded devices in some embodiments of this disclosure can first receive the initial position information set corresponding to the target object set in the image to be detected sent by the central processing unit. Based on the initial position information set, an initial size value set for the target object set is determined. Here, the obtained initial position information set and initial size value set facilitate the subsequent determination of the first threshold size value and the second threshold size value of the feature space size scaling layer. Secondly, the first threshold size and the second threshold size corresponding to the input feature space of the feature space scaling layer in the classification neural network are determined. The classification neural network also includes a feature extraction layer and a fully connected layer. Determining the first threshold size avoids inputting sub-image sets smaller than the first threshold size to the feature extraction layer, which could cause errors in the classification neural network. Determining the second threshold size facilitates the subsequent determination of the target size set. Next, a target size set is generated based on the first threshold size, the second threshold size, and the initial size set. This generated target size set avoids errors caused by sub-image sets smaller than the first threshold size. For sub-image sets larger than the second threshold size, it facilitates determining the method for generating the target size set and facilitates subsequent classification processing. Then, a subset of initial position information corresponding to the target size set is selected from the initial position information set as the target position information set. Determining the target position information set facilitates the subsequent determination of the sub-image set corresponding to the target position set and the image to be detected. Finally, the target position information set and the image to be detected are input to the feature extraction layer to obtain the first target feature space set. Here, by directly reading the sub-image set corresponding to the target location information set from the image to be detected, the storage and processing of the captured image by the central processing unit can be eliminated, reducing the load on the central processing unit and memory space, and improving the processing efficiency and stability of image detection and classification of embedded devices.Then, the first target feature space set is input to the feature space scaling layer to obtain a second target feature space set with a preset size. The feature space scaling layer is a neural network layer that constrains the input first target feature space set to obtain the second target feature space set with the preset size. Here, the feature space scaling layer can accept target feature spaces of different sizes, thereby eliminating the need for image cropping and scaling by the central processing unit (CPU), improving the efficiency and stability of target detection and classification, and reducing the transmission bandwidth between the CPU and the neural network processor. Finally, the second target feature space set is input to the fully connected layer to obtain the classification result set. Therefore, this target detection and classification method based on embedded devices optimizes the target detection and classification process on embedded devices by constraining the feature space set input to the feature space scaling layer, thereby improving the efficiency and stability of target detection and image classification.

[0073] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a target detection and classification device based on an embedded device. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the target detection and classification device based on embedded devices can be specifically applied to various electronic devices.

[0074] like Figure 2As shown, a target detection and classification device 200 based on an embedded device includes: a receiving unit 201, a first determining unit 202, a second determining unit 203, a generating unit 204, a filtering unit 205, a first input unit 206, a second input unit 207, and a third input unit 208. The receiving unit 201 is configured to receive an initial position information set corresponding to a set of target objects in an image to be detected, sent by a central processing unit. The first determining unit 202 is configured to determine an initial size value set for the target object set based on the initial position information set. The second determining unit 203 is configured to determine a first threshold size value and a second threshold size value corresponding to the input feature space of a feature space scaling layer included in a classification neural network, wherein the classification neural network further includes a feature extraction layer and a fully connected layer. The generating unit 204 is configured to generate a target size value set based on the first threshold size value, the second threshold size value, and the initial size value set. The filtering unit 205 is configured to filter out a subset of initial position information corresponding to the target size value set from the initial position information set, as the target position information set. The first input unit 206 is configured to input the target location information set and the image to be detected into the feature extraction layer to obtain a first target feature space set. The second input unit 207 is configured to input the first target feature space set into the feature space scaling layer to obtain a second target feature space set with a preset size value, wherein the feature space scaling layer is a neural network layer that constrains the input first target feature space set to obtain the second target feature space set with the preset size value. The third input unit 208 is configured to input the second target feature space set into the fully connected layer to obtain a classification result set.

[0075] It is understandable that the units and references described in the target detection and classification device 200 based on embedded devices... Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the target detection and classification device 200 based on embedded devices and the units contained therein, and will not be repeated here.

[0076] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0077] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0078] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0079] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0080] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0081] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0082] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive an initial position information set corresponding to a set of target objects in an image to be detected, sent by a central processing unit; determine an initial size value set for the target object set based on the initial position information set; determine a first threshold size value and a second threshold size value corresponding to the input feature space of a feature space scaling layer included in a classification neural network, wherein the classification neural network further includes a feature extraction layer and a fully connected layer; and generate a target size based on the first threshold size value, the second threshold size value, and the initial size value set. The initial position information set is selected from the initial position information set and a subset of initial position information corresponding to the target size value set is selected as the target position information set. The target position information set and the image to be detected are input into the feature extraction layer to obtain the first target feature space set. The first target feature space set is input into the feature space size scaling layer to obtain the second target feature space set with a preset size value. The feature space size scaling layer is a neural network layer that performs constraint processing on the input first target feature space set to obtain the second target feature space set with a preset size value. The second target feature space set is input into the fully connected layer to obtain the classification result set.

[0083] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a receiving unit, a first determining unit, a second determining unit, a generating unit, a filtering unit, a first input unit, a second input unit, and a third input unit. The names of these units do not necessarily limit the specific unit; for example, the receiving unit may also be described as "a unit that receives an initial position information set corresponding to a set of target objects in an image to be detected, sent by a central processing unit."

[0086] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0087] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A target detection and classification method based on embedded devices, comprising: Receive the image to be detected sent by the central processing unit; The image to be detected is input into the target detection model to obtain a detection result set; The detection result set is sent to the central processing unit; The system receives an initial location information set corresponding to a set of target objects in an image to be detected, sent by a central processing unit (CPU). The initial location information set is obtained through the following steps: the CPU decodes the detection result set to obtain a decoded detection result set; the CPU performs candidate box filtering on the decoded detection result set to obtain the initial location information set. The central processing unit is used to perform candidate box filtering on the decoded detection result set to obtain an initial location information set, including: Clustering is performed on the decoded detection result set to obtain a decoded detection result set corresponding to each target object in the image to be detected, which serves as multiple clustered decoded detection result sets. For each of the multiple clustering decoding detection result sets, the following determination steps are performed: The clustering decoding result with the highest confidence value is selected from the clustering decoding result set and used as the target clustering decoding result. The location information corresponding to each cluster-decoded detection result in the cluster-decoded detection result set is standardized to generate standardized location information, thus obtaining a standardized location information set. The standardized location information corresponding to the detection result after target clustering decoding is selected from the standardized location information set and used as the target standardized location information; Based on the target standardized location information and the standardized location information set, a distance value set is generated; The target distance value set is obtained by filtering out distance values ​​greater than or equal to a preset distance threshold from the set of distance values; Determine the average deviation between each of the coincident standardized location information in the coincident standardized location information set corresponding to the target distance value set and the target standardized location information; The sum of the standardized target location information and the average deviation value is determined as the updated target decoding detection result; The location information corresponding to the detection results of the multiple updated targets is determined as the initial location information set; Based on the initial position information set, determine the initial size value set for the target object set; The classification neural network includes a feature space scaling layer, which includes a feature space size scaling layer. The first threshold size value and the second threshold size value are determined for the input feature space. The classification neural network also includes a feature extraction layer and a fully connected layer. Generate a target size value set based on the first threshold size value, the second threshold size value, and the initial size value set; A subset of initial position information corresponding to the target size value set is selected from the initial position information set and used as the target position information set. The target location information set and the image to be detected are input into the feature extraction layer to obtain the first target feature space set; The first target feature space set is input to the feature space size scaling layer to obtain a second target feature space set with a preset size value. The feature space size scaling layer is a neural network layer that performs constraint processing on the input first target feature space set to obtain a second target feature space set with a preset size value. The second target feature space set is input into the fully connected layer to obtain the classification result set.

2. The method according to claim 1, wherein, The determination of the first threshold size value and the second threshold size value corresponding to the input feature space of the feature space size scaling layer of the classification neural network includes: The output size value corresponding to the output feature space of the feature space scaling layer is determined as the first threshold size value; The size value that is a preset integer multiple of the output size value is determined as the second threshold size value.

3. The method according to claim 2, wherein, The step of generating a target size value set based on the first threshold size value, the second threshold size value, and the initial size value set includes: In response to determining that there is an initial size value in the initial size value set that is greater than the second threshold size value, the initial size values ​​that are greater than the second threshold size value are determined as the first initial size value set; Each sub-image to be detected in the sub-image set to be detected corresponding to the first initial size value set is scaled to generate scaled sub-images to be detected, resulting in a scaled sub-image set to be detected. The sub-image set to be detected is the initial position information set corresponding to the first initial size value set, and is a sub-image set directly read from the image to be detected. The initial size values ​​that are greater than the first threshold size value and less than the second threshold size value are selected from the initial size value set and used as the second initial size values ​​to obtain the second initial size value set; The scaled size value set corresponding to the scaled sub-image set to be detected and the second initial size value set are determined as the target size value set.

4. A target detection and classification device based on an embedded device, comprising: The first receiving unit is configured to receive the image to be detected sent by the central processing unit; The first input unit is configured to input the image to be detected into the target detection model to obtain a detection result set; The sending unit is configured to send the detection result set to the central processing unit; The second receiving unit is configured to receive an initial position information set corresponding to a set of target objects in an image to be detected, sent by a central processing unit (CPU). The initial position information set is obtained through the following steps: the CPU decodes the detection result set to obtain a decoded detection result set; the CPU performs candidate box filtering on the decoded detection result set to obtain the initial position information set; wherein the CPU performs candidate box filtering on the decoded detection result set to obtain the initial position information set, including: clustering the decoded detection result set to obtain a decoded detection result set corresponding to each target object in the image to be detected, serving as multiple clustered decoded detection result sets; for each clustered decoded detection result set in the multiple clustered decoded detection result sets, the following determination step is performed: selecting the clustered decoded detection result with the highest confidence value from the clustered decoded detection result set as the target cluster. Decoded detection results; standardize the location information corresponding to each clustered decoding detection result in the clustered decoding detection result set to generate standardized location information, thus obtaining a standardized location information set; select the standardized location information corresponding to the target clustered decoding detection result from the standardized location information set as the target standardized location information; generate a distance value set based on the target standardized location information and the standardized location information set; select distance values ​​greater than or equal to a preset distance threshold from the distance value set to obtain the target distance value set; determine the average deviation value between each overlapping standardized location information in the overlapping standardized location information set corresponding to the target distance value set and the target standardized location information; determine the sum of the target standardized location information and the average deviation value as the updated target decoding detection result; determine the location information corresponding to the multiple updated target decoding detection results as the initial location information set; The first determining unit is configured to determine an initial set of size values ​​for the target object set based on the initial position information set. The second determining unit is configured to determine a first threshold size value and a second threshold size value corresponding to the input feature space of the feature space size scaling layer included in the classification neural network, wherein the classification neural network further includes: a feature extraction layer and a fully connected layer; The generation unit is configured to generate a target size value set based on the first threshold size value, the second threshold size value, and the initial size value set; The filtering unit is configured to filter out a subset of initial position information corresponding to the target size value set from the initial position information set, and use it as the target position information set. The second input unit is configured to input the target location information set and the image to be detected into the feature extraction layer to obtain the first target feature space set; The third input unit is configured to input the first target feature space set into the feature space size scaling layer to obtain a second target feature space set with a preset size value, wherein the feature space size scaling layer is a neural network layer that performs constraint processing on the input first target feature space set to obtain a second target feature space set with a preset size value; The fourth input unit is configured to input the second target feature space set into the fully connected layer to obtain a classification result set.

5. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3.

6. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

Images