Image processing method and device based on target detection model, and storage medium

Abstract

The invention discloses an image processing method and device based on a target detection model and a storage medium, and relates to the field of electrical digital data processing, and the method comprises the steps: transmitting a collected face detection image of a target object to the target detection model; extracting a target image feature vector of the face detection image by using the target detection model, weighting the target image feature vector, and predicting a target anchor frame in the face detection image based on the weighted target image feature vector, an object state corresponding to the target image feature vector is generated based on the target anchor box, and the target detection model is obtained by taking a historical face image of a target object as an input sample and taking a historical object state corresponding to the historical face image as an output sample for training; and determining the object intention of the target object according to the object state output by the target detection model.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing, and more specifically, to an image processing method, apparatus, and storage medium based on a target detection model. Background Technology

[0002] Currently, in various application scenarios of object detection, the detection of faces can determine a user's behavior or intent. For example, if a user scans their face to enter a train station or airport, it can be inferred that their behavior is to board a train or bus. However, face scanning is an example of detecting a large target. In a surveillance scenario, a face becomes a small target relative to the overall image, requiring the use of small target detection techniques to obtain crucial information.

[0003] In related technologies, although small target detection of users can be achieved through existing detection models, the backbone of existing detection models is a residual structure. When using the residual structure to extract features, the inherent deepening logic of the residual structure will cause the extracted feature information to become less and less related to the original information. This makes the detection of users inaccurate and unable to accurately determine the user's intent.

[0004] Therefore, in related technologies, there is a technical problem that the detection of users is not accurate enough, which makes it impossible to accurately determine the user's intention.

[0005] There is no effective solution yet to address the technical problem in related technologies where inaccurate user detection leads to an inability to accurately determine user intent. Summary of the Invention

[0006] This application provides an image processing method, apparatus, and storage medium based on a target detection model, to at least solve the technical problem in the related art where insufficient user detection leads to an inability to accurately determine the user's intent.

[0007] According to one embodiment of this application, an image processing method based on an object detection model is provided, comprising: sending a collected face detection image of a target object to an object detection model; using the object detection model to extract a target image feature vector from the face detection image, assigning weights to the target image feature vector, predicting a target anchor box in the face detection image based on the weighted target image feature vector, and generating an object state corresponding to the target image feature vector based on the target anchor box, wherein the object detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples; and determining the object intent of the target object based on the object state output by the object detection model.

[0008] In an exemplary embodiment, the target detection model includes a channel-level feature module. Extracting the target image feature vector of the face detection image using the target detection model includes: extracting the target image feature vector of the face detection image using the channel-level feature module, wherein the channel-level feature module is obtained by replacing the initial bottleneck cross-stage partial network layer of the initial channel-level feature module with the target bottleneck cross-stage partial network layer, and the channel-level feature module includes at least the target bottleneck cross-stage partial network layer.

[0009] In an exemplary embodiment, the initial channel-level feature module further includes a first convolutional layer, a first separation layer, and a first fusion layer. The first convolutional layer performs convolution operations on the face detection image in each image channel to obtain a result, and uses an activation function to generate a feature image corresponding to the face detection image based on the result. The first separation layer divides the feature image into a first feature image and a second feature image based on the image channel dimension of the face detection image. The first feature image corresponds to the first channel dimension, and the second feature image corresponds to the second channel dimension. The sum of the dimensions of the first channel dimension and the second channel dimension is the image channel dimension. The initial bottleneck cross-stage network layer has a classical residual structure. The classical residual structure extracts a first image feature vector from the first feature image and a second image feature vector from the second feature image, and obtains a summation result of the first image feature vector and the second image feature vector. The first fusion layer performs feature fusion on the summation result to obtain a first fusion result.

[0010] In an exemplary embodiment, the channel-level feature module further includes a second convolutional layer, a second separation layer, and a second fusion layer. The second convolutional layer is used to perform convolution operations on the face detection image in each image channel to obtain the operation result, and uses an activation function to generate a feature image corresponding to the face detection image based on the operation result. The second separation layer is used to divide the feature image into a third feature image and a fourth feature image based on the image channel dimension of the face detection image. The third feature image corresponds to the third channel dimension, the fourth feature image corresponds to the fourth channel dimension, and the sum of the dimensions of the third channel dimension and the fourth channel dimension is the image channel dimension.

[0011] In an exemplary embodiment, the target bottleneck cross-stage partial network layer includes: a first extraction layer and a second extraction layer for extracting features from the fourth feature image. Extracting the target image feature vector of the face detection image using the channel-level feature module includes: extracting the target image feature vector of the face detection image using the target bottleneck cross-stage partial network layer in the following manner: obtaining a third feature image output via the first extraction layer, and obtaining a third image feature vector output by the second extraction layer based on the fourth feature image; using the second fusion layer to perform feature fusion on the third feature image and the third image feature vector to obtain a second fusion result.

[0012] In an exemplary embodiment, obtaining the third image feature vector output by the second extraction layer based on the fourth feature image includes: performing feature extraction on the fourth feature image using different convolution kernels of the second extraction layer to obtain different extraction results; performing feature fusion operation on the feature vectors in the different extraction results to obtain a third fusion result, wherein the third fusion result includes the fourth image feature vector corresponding to the fourth feature image; and, if it is determined that the channel dimension of the fourth image feature vector is inconsistent with the fourth channel dimension, using an activation function to restore the channel dimension of the fourth image feature vector to the fourth channel dimension to obtain the third image feature vector.

[0013] In an exemplary embodiment, the channel-level feature module further includes a third convolutional layer, and the method further includes: using the third convolutional layer to perform a convolution operation on the feature vector in the second fusion result to obtain the target image feature vector of the face detection image.

[0014] In an exemplary embodiment, the target image feature vector is weighted using an attention mechanism algorithm in the following manner: for the i-th image channel dimension of the face detection image, the target image feature vector is vector-encoded using a first convolution kernel in the horizontal direction to obtain the i-th width vector corresponding to the target image feature vector; and the target image feature vector is vector-encoded using a second convolution kernel in the vertical direction to obtain the i-th height vector corresponding to the target image feature vector.

[0015] The i-th width vector is represented as follows:

[0016]

[0017] x∈R A×B×C A represents the image height, B represents the image width, C represents the color channel dimension of the image, and m represents the x-coordinate of the pixel in the feature image corresponding to the face detection image.

[0018] The i-th height vector is represented as follows:

[0019]

[0020] n represents the ordinate of a pixel in the feature image corresponding to the face detection image; the i-th width vector and the i-th height vector are concatenated to obtain a concatenated vector; a third convolution kernel is used to perform a convolution transformation operation on the i-th concatenated vectors to generate a first intermediate vector; an activation function is used to calculate the first intermediate vector to obtain a second intermediate vector; the second intermediate vector is split into a first spatial vector and a second spatial vector according to the spatial direction, wherein the first spatial vector corresponds to the horizontal direction and the second spatial vector corresponds to the vertical direction; the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector are used to assign weights to the feature vector of the target image.

[0021] In an exemplary embodiment, before assigning weights to the target image feature vector using the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector, the method further includes: performing a convolution transformation operation on the first spatial vector using the third convolution kernel to obtain a third intermediate vector; and calculating the first weight using an activation function on the third intermediate vector.

[0022] In an exemplary embodiment, before assigning weights to the target image feature vector using the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector, the method further includes: performing a convolution transformation operation on the second spatial vector using the third convolution kernel to obtain a fourth intermediate vector; and calculating the second weight using an activation function on the fourth intermediate vector.

[0023] In an exemplary embodiment, before generating the object state corresponding to the feature vector of the target image based on the target anchor box, the method further includes: obtaining the target anchor box by: performing a translational scan of the face detection image using an anchor box tool until all anchor boxes are marked in the face detection image, wherein each anchor box is of the same size; calculating the confidence score of each anchor box, wherein the confidence score represents the probability that each anchor box contains the object to be detected; and obtaining the target anchor box with the highest confidence score from all the anchor boxes.

[0024] In an exemplary embodiment, predicting a target anchor box in a face detection image based on the weighted target image feature vector includes: improving the boundary loss function of the target detection model in response to a prompt message indicating that the target image feature vector has been weighted; predicting the size and position of the object to be detected in the face detection image based on the improved boundary loss function to obtain an anchor box containing the object to be detected in the face detection image, and determining the anchor box containing the object to be detected as the target anchor box.

[0025] In one exemplary embodiment, improving the boundary loss function of the object detection model includes replacing the boundary loss function of the object detection model with the following improved loss function to improve the boundary loss function of the object detection model:

[0026] Loss_new=αL CIOU +(1-α)NWD,

[0027] Where Loss_new represents the improved loss function, L CIOU Let CIOU represent the loss function, NWD represent the loss function, and α be an adjustable parameter.

[0028] In an exemplary embodiment, the NWD loss function is expressed as follows:

[0029]

[0030] Where C1 is the number of categories in the dataset. It is a distance metric, N a express The modeled Gaussian distribution, N b express The Gaussian distribution modeled.

[0031] In an exemplary embodiment, determining the object intent of the target object based on the object state output by the target detection model includes: if the object state is determined to indicate the mood state of the target object, determining a preset prompt for the mood state; sending the prompt and a preset operation corresponding to the prompt to the target object; and if the target object agrees to the preset operation, determining that the object intent of the target object is to perform the preset operation.

[0032] In one exemplary embodiment, determining the object intent of the target object based on the object state output by the target detection model includes: determining suggested information preset for the health state when it is determined that the object state is used to indicate the health state of the target object; sending the suggested information to the target object; and determining the object intent of the target object based on the response information of the target object to the suggested information.

[0033] In an exemplary embodiment, the method further includes: when training other target detection models using historical face images and historical behavior images of the target object as input samples and historical object states corresponding to the historical face images as output samples, inputting the face detection images and object behaviors of the target object into the trained other target detection models, and determining the object intent of the target object based on the object states output by the trained other target detection models.

[0034] According to another aspect of the embodiments of this application, an image processing apparatus based on an object detection model is also provided, comprising: a sending module, configured to send a collected face detection image of a target object to an object detection model; a generating module, configured to use the object detection model to extract a target image feature vector from the face detection image, assign weights to the target image feature vector, predict a target anchor box in the face detection image based on the weighted target image feature vector, and generate an object state corresponding to the target image feature vector based on the target anchor box, wherein the object detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples; and a determining module, configured to determine the object intent of the target object based on the object state output by the object detection model.

[0035] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is configured to execute the above-described image processing method based on the target detection model at runtime.

[0036] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described image processing method based on the target detection model through the computer program.

[0037] In this embodiment, the collected face detection image of the target object is sent to a target detection model. Then, the target detection model extracts the target image feature vector from the face detection image, assigns weights to the target image feature vector, predicts the target anchor box in the face detection image based on the weighted target image feature vector, and generates an object state corresponding to the target image feature vector based on the target anchor box. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples. Next, the object intent of the target object is determined based on the object state output by the target detection model. By adopting the above technical solution, a target detection model can be used to detect face images, and then the object intent of the target object can be determined based on the object state output by the target detection model. This solves the technical problem that inaccurate detection of the user (i.e., the target object) leads to an inability to accurately determine the user's intent, thereby achieving accurate determination of the user's intent and improving the user experience. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the hardware environment for an image processing method based on a target detection model according to an embodiment of this application;

[0041] Figure 2 This is a flowchart of an image processing method based on a target detection model according to an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of the C2F module in related technologies;

[0043] Figure 4 This is a schematic diagram of the C2F module according to an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of a C2F module based on an attention mechanism according to an embodiment of this application;

[0045] Figure 6 This is a schematic diagram illustrating the principle of weighted assignment according to an embodiment of this application;

[0046] Figure 7 This is a structural block diagram of an image processing apparatus based on a target detection model according to an embodiment of this application. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "first," "second," etc., in the specification of this application and the above-mentioned terms are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] According to one aspect of the embodiments of this application, an image processing method based on an object detection model is provided. This image processing method based on an object detection model is widely used in whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned image processing method based on an object detection model can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0050] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0051] This embodiment provides an image processing method based on a target detection model, applied to the aforementioned terminal device. Figure 2 This is a flowchart of an image processing method based on an object detection model according to an embodiment of this application. The process includes the following steps:

[0052] Step S202: Send the collected face detection image of the target object to the target detection model;

[0053] Step S204: Use the target detection model to extract the target image feature vector of the face detection image, assign weights to the target image feature vector, predict the target anchor box in the face detection image based on the weighted target image feature vector, and generate the object state corresponding to the target image feature vector based on the target anchor box. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples.

[0054] The aforementioned target anchor boxes represent prior knowledge used to predict the target location of an object (e.g., in face detection, the target location is the location of the facial features). Each anchor box has a confidence score, representing the probability that the box contains the object. Generating the object state corresponding to the target image feature vector based on the target anchor boxes can be understood as determining the object state based on the facial feature state after determining that the anchor box contains facial features according to its confidence score. Specifically, if the anchor box's confidence score is greater than a threshold, the target location is confirmed to be contained. Determining the object state based on the facial feature state involves searching a database for object states that match the facial feature state.

[0055] For example, the target detection model may include a channel-level feature module, an attention mechanism module, a prediction module, and an output module. The channel-level feature module is used to extract the target image feature vector of the face detection image. The attention mechanism module is used to assign weights to the target image feature vector using an attention mechanism algorithm. The prediction module is used to predict the target anchor box in the face detection image based on the weighted target image feature vector, and to generate the object state corresponding to the target image feature vector based on the target anchor box. The output module is used to output the object state.

[0056] Optionally, if the object detection model is, for example, a variant of the YOLO model, then the object detection model may further include the following parts:

[0057] YOLO (You Only Look Once) is a deep learning algorithm for object detection. A key feature of the YOLO model is its ability to quickly detect multiple objects in an image in real time. It achieves fast and accurate object detection by dividing the image into a grid and predicting the category and location of objects that may exist in each grid.

[0058] The basic convolutional layer section typically includes convolutional layers, batch normalization layers, and ReLU activation functions, which can extract low-level features of the image.

[0059] The ReLU (Rectified Linear Unit) activation function is an intuitive non-linear activation function that outputs the given value when the input value is positive and 0 when the input value is negative. This activation function has advantages such as simple computation and fast training speed, and is therefore widely used in many deep learning models. However, it also has some problems, such as potentially causing neurons to "die" during training (i.e., the neuron's output remains 0). To address this issue, several improved ReLU activation functions have been proposed, such as Leaky ReLU and Parametric ReLU (PReLU).

[0060] Dark Forest Convolutional Part: This refers to a convolutional structure composed of multi-scale convolutional layers. This structure can capture features at different scales, which helps to detect targets of different sizes.

[0061] Multi-scale prediction section: Used to make predictions at multiple scales, which helps detect targets of different sizes.

[0062] The loss function includes object confidence loss, classification loss, and localization loss, which are used to measure the minimization of the difference between the model prediction and the actual label.

[0063] Post-processing includes thresholding and non-maximum suppression. Non-maximum suppression removes overlapping predicted boxes, retaining only the best predicted box.

[0064] Optionally, the detection accuracy of small target detection models can be further improved by reducing the number of model parameters.

[0065] Step S206: Determine the object intent of the target object based on the object state output by the target detection model.

[0066] For example, the process of determining the object intent of the target object based on the object state output by the target detection model can be understood as detecting the face to obtain data such as the user's current mood and health status to predict the user's intent. Alternatively, it can be further combined with user behavior to predict the user's intent, and then adjust home appliances based on the user's intent to provide the user with a better experience.

[0067] Through the above steps, the collected face detection image of the target object is sent to the target detection model; then, the target detection model is used to extract the target image feature vector of the face detection image, and the target image feature vector is weighted. Based on the weighted target image feature vector, the target anchor box in the face detection image is predicted, and the object state corresponding to the target image feature vector is generated based on the target anchor box. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples. Then, the object intent of the target object is determined based on the object state output by the target detection model. Using the above technical solution, the target detection model can be used to detect face images, and then the object intent of the target object can be determined based on the object state output by the target detection model. This solves the technical problem that inaccurate user detection leads to an inability to accurately determine user intent, thereby achieving accurate determination of user intent and improving user experience.

[0068] In an exemplary embodiment, the target detection model includes a channel-level feature module. The process of using the target detection model to extract the target image feature vector of the face detection image includes: using the channel-level feature module to extract the target image feature vector of the face detection image, wherein the channel-level feature module is obtained by replacing the initial bottleneck cross-stage partial network layer of the initial channel-level feature module with the target bottleneck cross-stage partial network layer, and the channel-level feature module includes at least the target bottleneck cross-stage partial network layer.

[0069] In this embodiment, the network architecture of the target bottleneck cross-stage partial network layer is superior to that of the initial bottleneck cross-stage partial network layer, and the target bottleneck cross-stage partial network layer has a higher ability to extract feature vectors than the initial bottleneck cross-stage partial network layer. This embodiment obtains a channel-level feature module by replacing the original initial bottleneck cross-stage partial network layer of the initial channel-level feature module with the target bottleneck cross-stage partial network layer, thus completing the update process of the initial channel-level feature module. Optimizing the initial channel-level feature module can improve its ability to extract feature vectors.

[0070] Furthermore, in an exemplary embodiment, the initial channel-level feature module further includes a first convolutional layer, a first separation layer, and a first fusion layer. The first convolutional layer performs convolution operations on the face detection image in each image channel to obtain the operation result, and uses an activation function to generate a feature image corresponding to the face detection image based on the operation result. The first separation layer divides the feature image into a first feature image and a second feature image based on the image channel dimension of the face detection image. The first feature image corresponds to the first channel dimension, and the second feature image corresponds to the second channel dimension. The sum of the dimensions of the first channel dimension and the second channel dimension is the image channel dimension. The initial bottleneck cross-stage network layer has a classical residual structure. The classical residual structure is used to extract the first image feature vector of the first feature image and the second image feature vector of the second feature image, and to obtain the sum of the first image feature vector and the second image feature vector. The first fusion layer performs feature fusion on the summation result to obtain a first fusion result.

[0071] In one embodiment, taking a YOLO model based on the YOLOv8 algorithm as an example, the BottleneckCSP part (i.e., the initial bottleneck cross-stage network layer mentioned above) in the C2F (Channelwise Feature Module) of the model is reconstructed to enhance the effective channel feature weights and improve the ability to acquire multi-scale detailed features. By adding a coordinate attention mechanism to the output of the C2F module, the ability to locate user feature information is also improved. Finally, a new loss function based on the original loss function CIoU of YOLOv8 is proposed, which improves the accuracy and performance of small object detection.

[0072] For example, it can be combined Figure 3 The initial channel-level feature modules are described, such as... Figure 3 As shown, the traditional C2F module of the YOLO model includes a first convolutional layer, a first splitting layer, and a backbone channel-level feature module. Figure 3The BottleneckCSP (i.e., the initial bottleneck cross-stage network layer mentioned above), is the first fusion layer.

[0073] CSP stands for Cross-Stage Partial Network. BottleneckCSP's Bottleneck structure is a classic residual structure, which combines information from two parts through summation before passing it down. However, precisely because the network deepens, the semantic information of the newly generated feature map becomes increasingly less related to the initial feature map.

[0074] BottleneckCSP is a network architecture designed to improve the computational efficiency and feature extraction capabilities of a model. This component is used in the backbone of the network, specifically the feature extraction stage. For example, in the YOLOv8 model, BottleneckCSP serves as the basic module for feature extraction. By stacking multiple BottleneckCSP modules to construct the network's backbone, the model's accuracy in object detection tasks can be improved while maintaining real-time performance.

[0075] The BottleneckCSP component has the following functions:

[0076] 1. By introducing a bottleneck structure, the number of model parameters can be reduced while maintaining performance.

[0077] 2. The Bottleneck CSP architecture helps reduce the computational cost of the model, thereby improving the model's running speed.

[0078] 3. The CSP (Cross Stage Partial) structure allows features from different stages to be fused, which helps to extract richer feature information.

[0079] 4. The Bottleneck CSP structure has a certain regularization effect, which can improve the generalization ability of the model.

[0080] It's important to note that YOLOv8 is not the official version of the YOLO series, so the specific network structure and component implementations may differ. However, BottleneckCSP, as an effective network architecture design, has been widely used in many modern object detection models, including YOLO and SSD (Single Shot MultiBox Detector).

[0081] It is understandable that a C2F (Channelwise Feature Module) module represents a specific convolutional layer structure.

[0082] In neural networks, convolutional layers, such as basic convolutional layers, can extract image features. Specifically, this is achieved by using a set of learnable filters (or convolutional kernels) to perform a sliding window calculation on the input data, generating feature maps. These feature maps can then be used in other layers, such as pooling layers and activation layers. However, basic convolutional layers have limitations: their kernels only focus on local regions of the input data and share the same weights across the entire input. This results in feature vectors extracted by basic convolutional layers being less accurate.

[0083] The C2F module focuses on handling the information exchange between different channels. In deep learning and computer vision, this involves feature extraction and processing performed on a channel-by-channel basis. It's important to note that in neural network architectures, especially convolutional neural networks (CNNs), channels typically represent different features or color information of an image. The channel-level feature module aims to optimize and improve the extraction and representation of these features, thereby enhancing the model's accuracy and performance.

[0084] The C2F module can perform the following operations:

[0085] Separate channels: Separate each channel of the input feature map and apply specific operations to each channel, such as convolution and normalization.

[0086] Channel merging: The processed channels are recombined to form a new output feature map.

[0087] It is evident that the C2F module focuses more on interaction and information integration at the channel level, and can improve the effect of feature extraction by processing channel information in a more detailed manner.

[0088] Furthermore, in the YOLOv8 model, the concat operation after Bottleneck CSP can be categorized as a feature fusion part. "Concat" is an abbreviation for "concatenation," which stands for "feature splicing" or "feature connection." In neural networks, feature splicing is a common operation used to combine feature maps from different layers or resolutions to achieve richer feature representations. In other words, the concat operation can splice feature maps from different layers or resolutions to achieve feature fusion. The concat operation has the following functions:

[0089] 1. Feature fusion: The concat operation stitches feature maps from different stages together, which helps to fuse multi-scale feature information in subsequent network layers.

[0090] 2. Information Complementarity: Feature maps from different layers may capture different aspects of information from the input image. Through the concat operation, this information can complement each other, improving the model's ability to recognize and locate targets.

[0091] 3. Improve robustness: Fusing multi-scale features can improve the model's ability to detect targets of different sizes, thereby enhancing the model's robustness.

[0092] 4. Network flexibility: The concat operation provides a flexible way to allow the model to introduce and fuse features at different stages, giving more freedom to network design.

[0093] In YOLOv8 or other YOLO-based models, feature concatenation operations are performed in the latter part of the model network, especially in Feature Pyramid Network (FPN) structures, to construct multi-scale feature representations. This multi-scale feature representation is crucial for object detection tasks, effectively improving the model's ability to detect objects of varying sizes.

[0094] In an exemplary embodiment, the channel-level feature module further includes a second convolutional layer, a second separation layer, and a second fusion layer. The second convolutional layer is used to perform convolution operations on the face detection image in each image channel to obtain the operation result, and uses an activation function to generate a feature image corresponding to the face detection image based on the operation result. The second separation layer is used to divide the feature image into a third feature image and a fourth feature image based on the image channel dimension of the face detection image. The third feature image corresponds to the third channel dimension, the fourth feature image corresponds to the fourth channel dimension, and the sum of the dimensions of the third channel dimension and the fourth channel dimension is the image channel dimension.

[0095] In an exemplary embodiment, the target bottleneck cross-stage partial network layer includes: a first extraction layer and a second extraction layer for extracting features from the fourth feature image. Extracting the target image feature vector of the face detection image using the channel-level feature module includes: extracting the target image feature vector of the face detection image using the target bottleneck cross-stage partial network layer in the following manner: obtaining a third feature image output via the first extraction layer, and obtaining a third image feature vector output by the second extraction layer based on the fourth feature image; using the second fusion layer to perform feature fusion on the third feature image and the third image feature vector to obtain a second fusion result.

[0096] In this embodiment, combined with Figure 4 This section explains the reconstructed BottleneckCSP portion (i.e., the target bottleneck cross-stage network layer) in the C2F module.

[0097] like Figure 4 As shown, let the input image size be A×B×C. The input image is separated into two parts, X1 and X2, along the channel dimension (C), which are A×B×p and A×B×q, respectively, where p+q=C. The output process for part X1 corresponds to the first extraction layer mentioned above; that is, no processing is done on X1, and it is directly output to the first fusion layer. Part X2 is the part where small targets are processed, and the processing of X2 corresponds to the second extraction layer mentioned above.

[0098] By performing a concat operation on the unprocessed X1 part and the processed X2 part, fused data (i.e. the second fusion result mentioned above) is obtained. Then, a 2×2 convolution operation is performed on the fused data to extract useful information from the X2 part while reducing redundant computation.

[0099] It should be noted that the image dimensions above are represented as A, B, and C, which is a representation method applicable to the fields of image processing and computer vision.

[0100] A represents the image height: the number of rows in the image, that is, the number of pixels in the image from top to bottom.

[0101] B represents the width: the number of columns in the image, that is, the number of pixels in the image from left to right.

[0102] C represents the number of image channels: the number of color dimensions in the image.

[0103] The number of channels represents the richness of color information stored in an image, and this concept is very important in image processing. For color images, a pixel typically consists of three color channels: red, green, and blue, so the number of channels for a color image is usually 3. For grayscale images, a pixel has only one brightness value and no color information, so the number of channels is 1.

[0104] For an RGBA image, in addition to the three color channels RGB, there is also an alpha channel used to represent the transparency of pixels. Therefore, an RGBA image has 4 channels.

[0105] In deep learning and convolutional neural networks, the number of channels in the input image directly impacts the network's structure and performance. For example, when processing an image, a convolutional layer performs convolution operations on each channel separately, and then generates a new feature map through an activation function. Therefore, increasing the number of channels increases the model's parameters and computational cost, but it also improves the model's ability to capture image features.

[0106] In an exemplary embodiment, the acquisition of the third image feature vector output by the second extraction layer based on the fourth feature image is further described as follows: different convolution kernels of the second extraction layer are used to extract features from the fourth feature image to obtain different extraction results; a feature fusion operation is performed on the feature vectors in the different extraction results to obtain a third fusion result, wherein the third fusion result includes the fourth image feature vector corresponding to the fourth feature image; if it is determined that the channel dimension of the fourth image feature vector is inconsistent with the fourth channel dimension, an activation function is used to restore the channel dimension of the fourth image feature vector to the fourth channel dimension to obtain the third image feature vector.

[0107] like Figure 4 As shown, when performing convolution operations on X2, 3×3, 4×4, and 5×5 convolution kernels (i.e., different convolution kernels in the second extraction layer mentioned above) can be used to extract image features, so as to obtain small target information of different sizes at different scales (i.e., different extraction results mentioned above). Then, feature fusion is performed on these small target information, so that all sub-features are aggregated and restored to the original dimensional order. Then, the number of channels is restored to p (i.e., the fourth channel dimension mentioned above) by using an activation function (e.g., the ReLU function or the Sigmoid function), to obtain X′2 (i.e., the third image feature vector mentioned above), which can be expressed by the following formula:

[0108] X'2=β(f Conv3x3 (X2)+f Conv4x4 (X2)+f Conv5x5 (X2)).

[0109] Where β represents the activation function, f Conv3x3 (X2) indicates that a 3x3 convolution kernel is applied to X2 to extract the feature vector.

[0110] Output = f Conv2x2 (Concat(X1,X'2)).

[0111] Concat(X1,X'2) represents re-splicing the two channels.

[0112] Taking an M×M convolutional kernel as an example, for the feature map A×B×C (i.e., the feature map corresponding to the image to be detected above), for the initial channel-level feature module containing the initial bottleneck cross-stage part of the network layer, the number of parameters and computational cost required to use an M×M convolutional kernel are as follows:

[0113] Parameter quantity: Parameter = M × M × C 2 .

[0114] Computational load: FLOPs = A × B × M × M × C 2 .

[0115] For channel-level feature modules that include network layers that cross-stage target bottlenecks, the number of parameters and computational complexity are as follows:

[0116]

[0117] Where p represents the proportion of channels performing ordinary convolution (X1 part) among all channels, and q represents the proportion of channels performing special convolution (corresponding to X2 part) among all channels. In other words, p and q are used to distinguish the proportion of channels performing different convolution calculations. For example, if q = p, it means that 1 / 2 of the channels are used for special convolution, and the remaining 1 / 2 of the channels are used for ordinary convolution. In this case, the number of parameters and computational cost of special convolution can be reduced by 1 / 4.

[0118] In scenarios such as smart home environments where simple structures, low latency, and large amounts of useless data are required to extract effective information, there is often a large amount of similar redundant background information and the target objects have large differences in scale. If a deep network model is used to extract features (corresponding to the initial channel-level feature module mentioned above), it will easily lead to gradient explosion. Therefore, dividing the channels and setting only some channels to perform special convolutions for feature extraction is beneficial to improving the overall smoothness of the network, enhancing the feature extraction capability for small targets, and making it more suitable for model deployment.

[0119] Based on the above embodiments, this application reconstructs the BottleneckCSP part in the C2F module based on the YOLOv8 algorithm, which can enhance the feature weights extracted from effective channels, thereby improving the ability to acquire multi-scale detailed features.

[0120] In an exemplary embodiment, the channel-level feature module further includes a third convolutional layer, and the method further includes: using the third convolutional layer to perform a convolution operation on the feature vector in the second fusion result to obtain the target image feature vector of the face detection image.

[0121] like Figure 4 As shown, in the concatenated feature map, since the X1 part has not been processed, there is still some useful local feature information. Therefore, Conv1x1 is finally used to improve the non-linear fitting ability of the model and to better learn the features of small targets.

[0122] In an exemplary embodiment, the target image feature vector is weighted using an attention mechanism algorithm in the following manner: For the i-th image channel dimension of the face detection image, a first convolutional kernel is used to vectorize the target image feature vector in the horizontal direction to obtain the i-th width vector corresponding to the target image feature vector; and a second convolutional kernel is used to vectorize the target image feature vector in the vertical direction to obtain the i-th height vector corresponding to the target image feature vector; wherein, the i-th width vector is represented as follows:

[0123]

[0124] x∈R A×B×C A represents the image height, B represents the image width, C represents the color channel dimension of the image, and m represents the x-coordinate of the pixel in the feature image corresponding to the face detection image.

[0125] The i-th height vector is represented as follows:

[0126]

[0127] n represents the ordinate of a pixel in the feature image corresponding to the face detection image;

[0128] The i-th width vector and the i-th height vector are concatenated to obtain a concatenated vector. A third convolution kernel is used to perform a convolution transformation on the i-th concatenated vectors to generate a first intermediate vector. An activation function is used to calculate the first intermediate vector to obtain a second intermediate vector. The second intermediate vector is then split into a first spatial vector and a second spatial vector according to spatial direction, where the first spatial vector corresponds to the horizontal direction and the second spatial vector corresponds to the vertical direction. The first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector are used to weight the target image feature vector. Here, i is a positive integer.

[0129] In this embodiment, combined with Figure 5 and Figure 6 Explain the empowerment process. For example... Figure 5 As shown, an attention mechanism is added to the feature vector output by the C2F module. Attention mechanism (AM) is a technique in computer science and deep learning used to enhance the model's focus on certain parts of the input data. Examples include cross-attention (CA) mechanisms. This attention mechanism can handle two input sequences, such as in machine translation or text summarization tasks. Accurate answer.

[0130] Next, as Figure 6 As shown, convolution kernels of (A, 1) (i.e., the first convolution kernel mentioned above) and (1, B) (i.e., the second convolution kernel mentioned above) are used to encode each channel in both the horizontal and vertical directions. In the i-th channel (i∈[1,C]), the height of the i-th channel is... (i.e., the i-th height vector mentioned above) and the width of the i-th channel (That is, the i-th width vector mentioned above) is shown below:

[0131]

[0132] Where, x∈R A×B×C , where m and n represent the x-coordinate and y-coordinate of a pixel in the feature map, respectively.

[0133] like Figure 6 As shown, a pair of z A and z B Feature concatenation is performed, and then a 1x1 convolution kernel (i.e., the third convolution kernel) is used to perform a convolution transformation to generate F1 (i.e., the first intermediate vector mentioned above). Then, an activation function is applied to F1 to generate f (i.e., the second intermediate vector), which is represented as follows:

[0134]

[0135] in, It is a splicing operation along the spatial dimension, where r represents the scaling factor for channel downsampling, and σ is a non-linear activation function.

[0136] Next, we split f into two tensors along different spatial dimensions, where is the tensor along the horizontal direction. (i.e., the first spatial vector mentioned above), with the vertical direction being... (i.e., the second spatial vector mentioned above).

[0137] This embodiment improves the ability to locate feature vectors by adding a coordinate attention mechanism to the output of the C2F module, thereby further improving the detection accuracy of the model.

[0138] In an exemplary embodiment, before assigning weights to the target image feature vector using the first weights corresponding to the first spatial vector and the second weights corresponding to the second spatial vector, the first spatial vector can be subjected to a convolution transformation operation using the third convolution kernel to obtain a third intermediate vector; and the first weights can be obtained by calculating the third intermediate vector using an activation function.

[0139] In an exemplary embodiment, before assigning weights to the target image feature vector using the first weights corresponding to the first spatial vector and the second weights corresponding to the second spatial vector, the second spatial vector can be subjected to a convolution transformation operation using the third convolution kernel to obtain a fourth intermediate vector; and the fourth intermediate vector can be calculated using an activation function to obtain the second weights.

[0140] For example, such as Figure 6 As shown, the process of obtaining the first and second weights can be expressed as follows: After obtaining the tensor... and Then, a 1x1 convolution kernel is used to apply f. A and f B Perform transformation operations to generate F respectively. A and F B Then for F A and F B Using the activation function, we obtain the attention weights w in the two spatial directions. A (i.e., the first weight mentioned above), w B (i.e., the second weight mentioned above): w A =σ(F A (f A )), w B =σ(F B (f B )).

[0141] Furthermore, the weighting of the target image feature vector using the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector can be expressed by the following formula:

[0142]

[0143] Among them, y i This represents the feature vector re-output based on weights, x∈R A×B×C , where m and n represent the x-coordinate and y-coordinate of a pixel in the feature map, respectively.

[0144] In an exemplary embodiment, further, before generating the object state corresponding to the feature vector of the target image based on the target anchor box, the method further includes: obtaining the target anchor box by: performing a translational scan of the face detection image using an anchor box tool until all anchor boxes are marked in the face detection image, wherein each anchor box is of the same size; calculating the confidence score of each anchor box, wherein the confidence score represents the probability that each anchor box contains the object to be detected; and obtaining the target anchor box with the highest confidence score from all the anchor boxes.

[0145] Optionally, a confidence score can be used to represent the probability that the predicted anchor box contains the target to be detected. The confidence score ranges from 0 to 1, with a higher value indicating higher confidence in the prediction. In practical applications, a confidence score threshold can be set to filter out predictions with low confidence scores, thereby reducing false detections.

[0146] In an exemplary embodiment, predicting a target anchor box in a face detection image based on the weighted target image feature vector includes: improving the boundary loss function of the target detection model in response to a prompt message indicating that the target image feature vector has been weighted; predicting the size and position of the object to be detected in the face detection image based on the improved boundary loss function to obtain an anchor box containing the object to be detected in the face detection image, and determining the anchor box containing the object to be detected as the target anchor box.

[0147] In an exemplary embodiment, a further technical solution for improving the boundary loss function of the object detection model is provided, which may include: replacing the boundary loss function of the object detection model with the following improved loss function to improve the boundary loss function of the object detection model: Loss_new = αL CIOU +(1-α)NWD, where Loss_new represents the improved loss function, L CIOU Let CIOU represent the loss function, NWD represent the loss function, and α is an adjustable parameter with a value range of [0,1].

[0148] The CIOU (Circle Intersection over Union) loss function is a loss function used for object detection tasks. It combines IoU (Intersection over Union) with other geometric factors such as center point distance and aspect ratio. The CIOU loss function aims to more effectively optimize the performance of object detectors, especially in terms of bounding box accuracy.

[0149] The NWD (Normalized Wasserstein Distance) loss function represents the loss function based on the normalized Wasserstein distance. NWD is commonly used in generative models, particularly variational autoencoders (VAEs) and other probability distribution-based generative models. The normalized Wasserstein distance is a method of measuring the difference between two probability distributions, taking into account the displacement and shape differences between the distributions. The NWD loss function trains the generative model by minimizing this distance, making its generated samples closer to the real data distribution.

[0150] This embodiment proposes a loss function that combines CIOU and NWD to improve the detection accuracy of the object detection model. NWD is a location regression loss function based on Wasserstein distance, which can use a two-dimensional Gaussian distribution to calculate the similarity between the predicted bounding box and the labeled bounding box. This method is insensitive to targets of different scales.

[0151]

[0152]

[0153] Where C1 is the number of categories in the dataset. It is a distance metric, N a and N b They represent and The model uses a Gaussian distribution. Since CIOU is more suitable for large targets and NWD is suitable for small targets, the recognition accuracy of small targets can be adjusted by using α according to the usage scenario.

[0154] While the CIOU loss function can be used directly, as shown in the formula above, v only reflects the aspect ratio difference and does not reflect the actual difference between the width and height of the detected target and its confidence score. This leads to low accuracy in the similarity between the learned predicted bounding boxes and the ground truth boxes. In other words, using only the CIOU loss function cannot significantly improve the accuracy and performance of small target detection. Furthermore, in real-world applications, since faces (or user behavior) are relatively rare in homes, an imbalanced training sample can occur, meaning that high-quality bounding boxes are few and low-quality (excessively large gradient) bounding boxes are many in an image, thus affecting the model's training performance.

[0155] It should be noted that, for the CIOU loss function, this application considers the overlapping area, center point distance, and aspect ratio of the bounding box regression, and uses the following formula to express the loss function formula:

[0156]

[0157] Where IOU represents the intersection-union ratio, C 2 It is the diagonal distance of the smallest bounding rectangle, ρ 2 (b,b gt ) represents the center point b of the predicted bounding box and the center point b of the ground truth bounding box. gt The Euclidean distance, where α is the equilibrium parameter, is expressed as follows:

[0158]

[0159] Where v represents the parameter with consistent aspect ratio, expressed as:

[0160]

[0161] in, and These are the aspect ratios of the ground truth bounding box and the predicted bounding box, respectively.

[0162] Through the above embodiments, this application first reconstructs the BottleneckCSP part in the C2F module based on the YOLOv8 algorithm, enhancing the effective channel feature weights and improving the ability to acquire multi-scale detailed features. Then, a coordinate attention mechanism is added to the output of the C2F module to improve the algorithm's ability to locate user feature information. Finally, a new loss function is proposed based on the original loss function CIoU of YOLOv8, improving the accuracy and performance of small target detection.

[0163] In an exemplary embodiment, the method for determining the object intent of the target object based on the object state output by the target detection model may include:

[0164] Option 1: If it is determined that the object state is used to indicate the mood state of the target object, a preset prompt message for the mood state is determined; the prompt message and the preset operation corresponding to the prompt message are sent to the target object; if it is determined that the target object agrees to the preset operation, the object intention of the target object is determined to perform the preset operation.

[0165] For example, if the target user is in a sad mood, the above prompt might be a comforting message. The corresponding preset action is to play soothing music. If the user agrees, it is determined that the user's intention is to play soothing music.

[0166] Option 2: If the object state is determined to indicate the health status of the target object, a preset suggestion message for the health status is determined; the suggestion message is sent to the target object; and the target object's object intent is determined based on the target object's response to the suggestion message.

[0167] For example, if the target individual's health status is sub-healthy, specifically such as slightly high blood pressure, the suggested information would be to recommend a low-salt diet. If the user's response is to schedule a physical examination, then it is determined that the user's intention is to undergo a physical examination.

[0168] Option 3 involves training other target detection models using historical face images and historical behavior images of the target object as input samples and historical object states corresponding to the historical face images as output samples. In this case, the face detection images and object behaviors of the target object are input into the trained other target detection models, and the object intent of the target object is determined based on the object states output by the trained other target detection models.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0170] Figure 7 This is a structural block diagram of an image processing apparatus based on a target detection model according to an embodiment of this application, such as... Figure 7 As shown, it includes:

[0171] The sending module 72 is used to send the collected face detection image of the target object to the target detection model;

[0172] The generation module 74 is used to extract the target image feature vector of the face detection image using the target detection model, assign weights to the target image feature vector, predict the target anchor box in the face detection image based on the weighted target image feature vector, and generate the object state corresponding to the target image feature vector based on the target anchor box. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples.

[0173] The determination module 76 is used to determine the object intent of the target object based on the object state output by the target detection model.

[0174] The aforementioned device sends the collected face detection image of the target object to a target detection model. The target detection model then extracts the target image feature vector from the face detection image, assigns weights to the feature vector, predicts target anchor boxes in the face detection image based on the weighted feature vector, and generates an object state corresponding to the target image feature vector based on the target anchor boxes. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to those historical face images as output samples. Finally, the object intent of the target object is determined based on the object state output by the target detection model. This technical solution allows for the detection of face images using a target detection model, and the determination of the target object's object intent based on the object state output by the target detection model. This solves the technical problem of inaccurate user detection leading to inaccurate determination of user intent, thereby achieving accurate determination of user intent and improving user experience.

[0175] In an exemplary embodiment, the generation module is further configured to: extract a target image feature vector from the face detection image using the channel-level feature module, wherein the channel-level feature module is obtained by replacing the initial bottleneck cross-stage partial network layer of the initial channel-level feature module with a target bottleneck cross-stage partial network layer, and the channel-level feature module includes at least a target bottleneck cross-stage partial network layer.

[0176] In an exemplary embodiment, the initial channel-level feature module further includes a first convolutional layer, a first separation layer, and a first fusion layer. The first convolutional layer performs convolution operations on the face detection image in each image channel to obtain a result, and uses an activation function to generate a feature image corresponding to the face detection image based on the result. The first separation layer divides the feature image into a first feature image and a second feature image based on the image channel dimension of the face detection image. The first feature image corresponds to the first channel dimension, and the second feature image corresponds to the second channel dimension. The sum of the dimensions of the first channel dimension and the second channel dimension is the image channel dimension. The initial bottleneck cross-stage network layer has a classical residual structure. The classical residual structure extracts a first image feature vector from the first feature image and a second image feature vector from the second feature image, and obtains a summation result of the first image feature vector and the second image feature vector. The first fusion layer performs feature fusion on the summation result to obtain a first fusion result.

[0177] In an exemplary embodiment, the channel-level feature module further includes a second convolutional layer, a second separation layer, and a second fusion layer. The second convolutional layer is used to perform convolution operations on the face detection image in each image channel to obtain the operation result, and uses an activation function to generate a feature image corresponding to the face detection image based on the operation result. The second separation layer is used to divide the feature image into a third feature image and a fourth feature image based on the image channel dimension of the face detection image. The third feature image corresponds to the third channel dimension, the fourth feature image corresponds to the fourth channel dimension, and the sum of the dimensions of the third channel dimension and the fourth channel dimension is the image channel dimension.

[0178] In an exemplary embodiment, the target bottleneck cross-stage partial network layer includes: a first extraction layer and a second extraction layer for extracting features from the fourth feature image. The generation module is further configured to: extract the target image feature vector of the face detection image using the target bottleneck cross-stage partial network layer in the following manner: obtain a third feature image output by the first extraction layer, and obtain a third image feature vector output by the second extraction layer based on the fourth feature image; and use the second fusion layer to perform feature fusion on the third feature image and the third image feature vector to obtain a second fusion result.

[0179] In an exemplary embodiment, the generation module is further configured to: extract features from the fourth feature image using different convolutional kernels of the second extraction layer to obtain different extraction results; perform feature fusion operation on the feature vectors in the different extraction results to obtain a third fusion result, wherein the third fusion result includes a fourth image feature vector corresponding to the fourth feature image; and, if it is determined that the channel dimension of the fourth image feature vector is inconsistent with the fourth channel dimension, use an activation function to restore the channel dimension of the fourth image feature vector to the fourth channel dimension to obtain the third image feature vector.

[0180] In an exemplary embodiment, the channel-level feature module further includes a third convolutional layer, and the generation module is further configured to: use the third convolutional layer to perform a convolution operation on the feature vector in the second fusion result to obtain the target image feature vector of the face detection image.

[0181] In an exemplary embodiment, the generation module is further configured to: weight the target image feature vector using an attention mechanism algorithm in the following manner: for the i-th image channel dimension of the face detection image, use a first convolution kernel to perform vector encoding on the target image feature vector in the horizontal direction to obtain the i-th width vector corresponding to the target image feature vector; and use a second convolution kernel to perform vector encoding on the target image feature vector in the vertical direction to obtain the i-th height vector corresponding to the target image feature vector;

[0182] The i-th width vector is represented as follows:

[0183]

[0184] x∈R A×B×C A represents the image height, B represents the image width, C represents the color channel dimension of the image, and m represents the x-coordinate of the pixel in the feature image corresponding to the face detection image.

[0185] The i-th height vector is represented as follows:

[0186]

[0187] n represents the ordinate of a pixel in the feature image corresponding to the face detection image; the i-th width vector and the i-th height vector are concatenated to obtain a concatenated vector; a third convolution kernel is used to perform a convolution transformation operation on the i-th concatenated vectors to generate a first intermediate vector; an activation function is used to calculate the first intermediate vector to obtain a second intermediate vector; the second intermediate vector is split into a first spatial vector and a second spatial vector according to the spatial direction, wherein the first spatial vector corresponds to the horizontal direction and the second spatial vector corresponds to the vertical direction; the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector are used to assign weights to the feature vector of the target image.

[0188] In an exemplary embodiment, the generation module is further configured to: perform a convolution transformation operation on the first spatial vector using the third convolution kernel to obtain a third intermediate vector; and calculate the first weight using an activation function on the third intermediate vector.

[0189] In an exemplary embodiment, the generation module is further configured to: perform a convolution transformation operation on the second spatial vector using the third convolution kernel to obtain a fourth intermediate vector; and calculate the second weight using an activation function on the fourth intermediate vector.

[0190] In an exemplary embodiment, the generation module is further configured to: obtain the target anchor box by: performing a translational scan on the face detection image using an anchor box tool until all anchor boxes are marked in the face detection image, wherein each anchor box is of the same size; calculating the confidence score of each anchor box, wherein the confidence score represents the probability that each anchor box contains the object to be detected; and obtaining the target anchor box with the highest confidence score from all the anchor boxes.

[0191] In an exemplary embodiment, the generation module is further configured to: improve the boundary loss function of the target detection model in response to a prompt message after weighting the feature vector of the target image; predict the size and position of the object to be detected in the face detection image based on the improved boundary loss function, obtain an anchor box containing the object to be detected in the face detection image, and determine the anchor box containing the object to be detected as the target anchor box.

[0192] In an exemplary embodiment, the generation module is further configured to: replace the boundary loss function of the object detection model with the following improved loss function to improve the boundary loss function of the object detection model:

[0193] Loss_new=αL CIOU +(1-α)NWD,

[0194] Where Loss_new represents the improved loss function, L CIOU Let CIOU represent the loss function, NWD represent the loss function, and α be an adjustable parameter.

[0195] In an exemplary embodiment, the NWD loss function is expressed as follows:

[0196]

[0197] Where C1 is the number of categories in the dataset. It is a distance metric, N a express The modeled Gaussian distribution, N b express The Gaussian distribution modeled.

[0198] In an exemplary embodiment, the determining module is further configured to: determine a preset prompt for the mood state when the object state is determined to indicate the mood state of the target object; send the prompt and a preset operation corresponding to the prompt to the target object; and determine that the target object's object intent is to perform the preset operation when the target object agrees to the preset operation.

[0199] In an exemplary embodiment, the determining module is further configured to: determine suggested information preset for the health status when it is determined that the object status is used to indicate the health status of the target object; send the suggested information to the target object; and determine the object intent of the target object based on the response information of the target object to the suggested information.

[0200] In an exemplary embodiment, the image processing apparatus based on the object detection model further includes a second determining module, configured to: when training other object detection models using historical face images and historical behavior images of the target object as input samples and historical object states corresponding to the historical face images as output samples, input the face detection images of the target object and the object behavior of the target object into the trained other object detection models, and determine the object intent of the target object based on the object states output by the trained other object detection models.

[0201] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0202] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0203] S1, send the collected face detection image of the target object to the target detection model;

[0204] S2, using the target detection model to extract the target image feature vector of the face detection image, assigning weights to the target image feature vector, predicting the target anchor box in the face detection image based on the weighted target image feature vector, and generating the object state corresponding to the target image feature vector based on the target anchor box. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples.

[0205] S3, determine the object intent of the target object based on the object state output by the target detection model.

[0206] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0207] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0208] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0209] S1, send the collected face detection image of the target object to the target detection model;

[0210] S2, using the target detection model to extract the target image feature vector of the face detection image, assigning weights to the target image feature vector, predicting the target anchor box in the face detection image based on the weighted target image feature vector, and generating the object state corresponding to the target image feature vector based on the target anchor box. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples.

[0211] S3, determine the object intent of the target object based on the object state output by the target detection model.

[0212] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0213] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0214] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0215] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method based on a target detection model, characterized in that, include: The collected face detection images of the target object are sent to the target detection model; The target detection model is used to extract the target image feature vector of the face detection image, and the target image feature vector is weighted. Based on the weighted target image feature vector, the target anchor box in the face detection image is predicted, and the object state corresponding to the target image feature vector is generated based on the target anchor box. The target detection model is trained by using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples. The object intent of the target object is determined based on the object state output by the target detection model.

2. The image processing method based on the target detection model according to claim 1, characterized in that, The target detection model includes a channel-level feature module. The target detection model is used to extract the target image feature vector of the face detection image, including: The channel-level feature module is used to extract the target image feature vector of the face detection image. The channel-level feature module is obtained by replacing the initial bottleneck cross-stage partial network layer of the initial channel-level feature module with the target bottleneck cross-stage partial network layer. The channel-level feature module includes at least the target bottleneck cross-stage partial network layer.

3. The image processing method based on the target detection model according to claim 2, characterized in that, The initial channel-level feature module further includes a first convolutional layer, a first splitting layer, and a first fusion layer. The first convolutional layer performs convolution operations on the face detection image in each image channel to obtain the operation result, and uses an activation function to generate a feature image corresponding to the face detection image based on the operation result. The first splitting layer divides the feature image into a first feature image and a second feature image based on the image channel dimension of the face detection image. The first feature image corresponds to the first channel dimension, and the second feature image corresponds to the second channel dimension. The sum of the dimensions of the first channel dimension and the second channel dimension is the image channel dimension. The initial bottleneck cross-stage network layer has a classical residual structure. The classical residual structure is used to extract the first image feature vector of the first feature image and the second image feature vector of the second feature image, and to obtain the sum of the first image feature vector and the second image feature vector. The first fusion layer performs feature fusion on the summation result to obtain a first fusion result.

4. The image processing method based on the target detection model according to claim 2, characterized in that, The channel-level feature module further includes a second convolutional layer, a second separation layer, and a second fusion layer. The second convolutional layer is used to perform convolution operations on the face detection image in each image channel to obtain the operation result, and uses an activation function to generate a feature image corresponding to the face detection image based on the operation result. The second separation layer is used to divide the feature image into a third feature image and a fourth feature image based on the image channel dimension of the face detection image. The third feature image corresponds to the third channel dimension, the fourth feature image corresponds to the fourth channel dimension, and the sum of the dimensions of the third channel dimension and the fourth channel dimension is the image channel dimension.

5. The image processing method based on the target detection model according to claim 4, characterized in that, The target bottleneck cross-stage network layer includes: a first extraction layer and a second extraction layer for extracting features from the fourth feature image, using the channel-level feature module to extract the target image feature vector of the face detection image, including: The target image feature vector of the face detection image is extracted using the target bottleneck cross-stage partial network layer in the following manner: Obtain the third feature image output by the first extraction layer, and obtain the third image feature vector output by the second extraction layer based on the fourth feature image; The second fusion layer is used to perform feature fusion on the third feature image and the feature vector of the third image to obtain a second fusion result.

6. The image processing method based on the target detection model according to claim 5, characterized in that, Obtaining the third image feature vector output by the second extraction layer based on the fourth feature image includes: Different extraction results are obtained by using different convolutional kernels of the second extraction layer to extract features from the fourth feature image; A feature fusion operation is performed on the feature vectors in the different extraction results to obtain a third fusion result, wherein the third fusion result includes the fourth image feature vector corresponding to the fourth feature image; If the channel dimension of the fourth image feature vector is determined to be inconsistent with the fourth channel dimension, an activation function is used to restore the channel dimension of the fourth image feature vector to the fourth channel dimension to obtain the third image feature vector.

7. The image processing method based on the target detection model according to claim 6, characterized in that, The channel-level feature module further includes a third convolutional layer, and the method further includes: The third convolutional layer is used to perform a convolution operation on the feature vector in the second fusion result to obtain the target image feature vector of the face detection image.

8. The image processing method based on the target detection model according to claim 1, characterized in that, The target image feature vector is weighted using an attention mechanism algorithm in the following manner: For the i-th image channel dimension of the face detection image, the first convolution kernel is used in the horizontal direction to perform vector encoding on the feature vector of the target image to obtain the i-th width vector corresponding to the feature vector of the target image; And in the vertical direction, a second convolution kernel is used to perform vector encoding on the feature vector of the target image to obtain the i-th height vector corresponding to the feature vector of the target image; The i-th width vector is represented as follows: x∈R A×B×C A represents the image height, B represents the image width, C represents the color channel dimension of the image, and m represents the x-coordinate of the pixel in the feature image corresponding to the face detection image. The i-th height vector is represented as follows: n represents the ordinate of a pixel in the feature image corresponding to the face detection image; The i-th width vector and the i-th height vector are concatenated to obtain the concatenated vector; The third convolution kernel is used to perform a convolution transformation operation on the i concatenated vectors to generate the first intermediate vector; The activation function is used to calculate the first intermediate vector to obtain the second intermediate vector; The second intermediate vector is split into a first spatial vector and a second spatial vector according to the spatial direction, wherein the first spatial vector corresponds to the horizontal direction and the second spatial vector corresponds to the vertical direction; the target image feature vector is weighted using the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector.

9. The image processing method based on the target detection model according to claim 8, characterized in that, Before assigning weights to the target image feature vector using the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector, the method further includes: The first spatial vector is convolved using the third convolution kernel to obtain the third intermediate vector. The first weight is obtained by calculating the third intermediate vector using an activation function.

10. The image processing method based on the target detection model according to claim 8, characterized in that, Before assigning weights to the target image feature vector using the first weight corresponding to the first spatial vector and the second weight corresponding to the second spatial vector, the method further includes: The third convolution kernel is used to perform a convolution transformation operation on the second spatial vector to obtain the fourth intermediate vector; The activation function is used to calculate the fourth intermediate vector to obtain the second weight.

11. The image processing method based on the target detection model according to claim 1, characterized in that, Before generating the object state corresponding to the feature vector of the target image based on the target anchor box, the method further includes: The target anchor frame is obtained using the following method: The face detection image is scanned by translating using the anchor box tool until all anchor boxes are marked in the face detection image, wherein each anchor box is the same size; Calculate the confidence level for each anchor frame, where the confidence level represents the probability that each anchor frame contains the object to be detected; Obtain the target anchor box with the highest confidence from all the anchor boxes.

12. The image processing method based on the target detection model according to claim 1, characterized in that, Predicting target anchor boxes in the face detection image based on the weighted target image feature vector includes: In response to the prompt message after weighting the feature vector of the target image, the boundary loss function of the target detection model is improved; Based on the improved boundary loss function, the size and position of the object to be detected in the face detection image are predicted to obtain the anchor box containing the object to be detected in the face detection image, and the anchor box containing the object to be detected is determined as the target anchor box.

13. The image processing method based on the target detection model according to claim 12, characterized in that, The boundary loss function of the target detection model is improved, including: The boundary loss function of the object detection model is replaced with the following improved loss function to improve the boundary loss function of the object detection model: Loss_new=αL CIOU +(1-α)NWD, Where Loss_new represents the improved loss function, L CIOU Let CIOU represent the loss function, NWD represent the loss function, and α be an adjustable parameter.

14. The image processing method based on the target detection model according to claim 13, characterized in that, The NWD loss function is expressed as follows: Where C1 is the number of categories in the dataset. It is a distance metric, N a express The modeled Gaussian distribution, N b express The Gaussian distribution modeled.

15. The image processing method based on the target detection model according to claim 1, characterized in that, Determining the object intent of the target object based on the object state output by the target detection model includes: If it is determined that the object state is used to indicate the mood state of the target object, a preset prompt message for the mood state is determined; Send the prompt and the corresponding preset operation to the target object; If it is determined that the target object agrees to the preset operation, the object intent of the target object is determined to perform the preset operation.

16. The image processing method based on the target detection model according to claim 1, characterized in that, Determining the object intent of the target object based on the object state output by the target detection model includes: If it is determined that the object state is used to indicate the health status of the target object, then suggested information is determined for the health status. The suggestion information is sent to the target object; The target object's intent is determined based on its response to the suggestion information.

17. The image processing method based on the target detection model according to claim 1, characterized in that, The method further includes: In the case of training other target detection models using historical face images and historical behavior images of the target object as input samples and historical object states corresponding to the historical face images as output samples, the face detection images and object behaviors of the target object are input into the trained other target detection models, and the object intent of the target object is determined based on the object states output by the trained other target detection models.

18. An image processing device based on a target detection model, characterized in that, include: The sending module is used to send the collected face detection image of the target object to the target detection model; The generation module is used to extract the target image feature vector of the face detection image using the target detection model, assign weights to the target image feature vector, predict the target anchor box in the face detection image based on the weighted target image feature vector, and generate the object state corresponding to the target image feature vector based on the target anchor box. The target detection model is trained using historical face images of the target object as input samples and historical object states corresponding to the historical face images as output samples. The determination module is used to determine the object intent of the target object based on the object state output by the target detection model.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 17.

Images