Image processing method and device, electronic equipment and computer readable storage medium

Abstract

An image processing method and device, electronic equipment and computer readable storage medium. The image processing method comprises: acquiring a detected image, the detected image comprising a target object; inputting the detected image into an image detection model to obtain an output result of the image detection model, the output result of the image detection model comprising a feature vector of the target object, the feature vector being used to identify the target object; the image detection model comprising M first sub-models and a second sub-model; inputting the detected image into the image detection model to obtain the output result of the image detection model, comprising: inputting the detected image into the M first sub-models respectively; inputting the results of N predetermined network layers corresponding to N first sub-models in the M first sub-models into the second sub-model respectively to obtain an output result of the second sub-model, the output result of the second sub-model comprising the feature vector of the target object. The method can save time and computing power, speed up detection, and meet the requirement of real-time.

Description

Technical Field

[0001] Embodiments of this disclosure relate to an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of computer technology, mobile phones, computers, and other electronic devices are increasingly integrated into people's lives and work, and their functions are becoming more and more diverse. In some scenarios, electronic devices need to detect feature information such as the user's face. For example, in a facial recognition check-in scenario, a user's facial image can be captured, facial feature information can be determined based on the image, and then the user can be identified based on the feature information to complete the check-in. Summary of the Invention

[0003] At least one embodiment of this disclosure provides an image processing method, comprising: acquiring a detected image, wherein the detected image includes a target object; inputting the detected image into an image detection model to obtain an output result of the image detection model, wherein the output result of the image detection model includes a feature vector of the target object, the feature vector being used to identify the target object; wherein the image detection model includes M first sub-models and second sub-models; inputting the detected image into the image detection model to obtain an output result of the image detection model includes: inputting the detected image into the M first sub-models respectively; inputting the results of N predetermined network layers corresponding to N of the M first sub-models into the second sub-model to obtain an output result of the second sub-model, wherein the output result of the second sub-model includes a feature vector of the target object; wherein M is a positive integer, and N is a positive integer less than or equal to M.

[0004] For example, in an image processing method provided in one embodiment of this disclosure, the target object is a face.

[0005] For example, an embodiment of the image processing method provided in this disclosure further includes: comparing the feature vector of the target object with multiple feature vectors in a database to determine a target feature vector from the multiple feature vectors that satisfies a distance condition with the feature vector of the target object, and using the label information corresponding to the target feature vector as the recognition result of the target object.

[0006] For example, in an image processing method provided in an embodiment of this disclosure, each of the M first sub-models includes a first input layer, a plurality of first intermediate layers and a first output layer connected in sequence, and the predetermined network layer corresponding to each of the N first sub-models is one of the plurality of first intermediate layers in each first sub-model or the first output layer in each first sub-model; the second sub-model includes a second input layer, a plurality of second intermediate layers and a second output layer connected in sequence.

[0007] For example, in an image processing method provided in one embodiment of this disclosure, the N predetermined network layers are layers of the results with the same dimension in the N first sub-models.

[0008] For example, in an image processing method provided in one embodiment of this disclosure, the plurality of first intermediate layers include a plurality of depthwise convolutional layers connected in sequence; or the plurality of first intermediate layers include a plurality of depthwise convolutional layers and a plurality of deconvolutional layers connected in sequence; the plurality of second intermediate layers include a plurality of depthwise convolutional layers connected in sequence; or the plurality of second intermediate layers include a plurality of depthwise convolutional layers and a plurality of deconvolutional layers connected in sequence.

[0009] For example, in an image processing method provided in one embodiment of this disclosure, inputting the results of N predetermined network layers corresponding to N of the M first sub-models into the second sub-model includes: performing calculations on the results of the N predetermined network layers to obtain calculation results; and inputting the calculation results into the second sub-model.

[0010] For example, in an image processing method provided in one embodiment of this disclosure, the operation includes multiplication or addition; wherein the results of the N predetermined network layers have the same dimension.

[0011] For example, in an image processing method provided in an embodiment of this disclosure, the output result of the image detection model further includes the output results of the M sub-models corresponding to the M first sub-models, wherein the output results of the M sub-models correspond to M types of features respectively.

[0012] For example, in an image processing method provided in one embodiment of this disclosure, the output results of the M sub-models include at least one of heatmap features, detection box features, and key point features.

[0013] For example, an embodiment of the image processing method provided in this disclosure further includes: acquiring multiple sample images and multiple label data corresponding to the multiple sample images, wherein each sample image includes a sample object; updating and iterating the parameters of an initial model based on the multiple sample images and the multiple label data until the training completion condition is met, and using the trained initial model as the image detection model.

[0014] For example, in an image processing method provided in an embodiment of this disclosure, the parameters of an initial model are updated and iterated based on the plurality of sample images and the plurality of label data, including: performing the following operations for each sample image: inputting the sample image into the initial model to obtain the output result of the initial model, wherein the output result of the initial model includes the initial feature vector of the sample object in the sample image; calculating loss information based on the output result of the initial model and the label data corresponding to the sample image; updating the parameters of the initial model based on the loss information; wherein the initial model includes M first initial sub-models and second initial sub-models; inputting the sample image into the initial model to obtain the output result of the initial model includes: inputting the sample image into the M first initial sub-models respectively; inputting the results of the N predetermined network layers corresponding to the N first initial sub-models in the M first initial sub-models into the second initial sub-model to obtain the output result of the second initial sub-model, wherein the output result of the second initial sub-model includes the initial feature vector of the sample object in the sample image.

[0015] For example, in an image processing method provided in an embodiment of this disclosure, the output result of the initial model further includes the output results of the M initial sub-models corresponding to the M first initial sub-models respectively; loss information is calculated based on the output results of the initial model and the label data corresponding to the sample image, including: using a loss function to perform operations on the output results of the M initial sub-models, the output results of the second initial sub-models and the label data to obtain the loss information.

[0016] At least one embodiment of this disclosure provides an image processing apparatus, including an acquisition module and a detection module. The acquisition module is configured to acquire a detected image, wherein the detected image includes a target object. The detection module is configured to input the detected image into an image detection model to obtain the output result of the image detection model, wherein the output result of the image detection model includes a feature vector of the target object, the feature vector being used to identify the target object. The image detection model includes M first sub-models and second sub-models. The detection module is further configured to: input the detected image into the M first sub-models respectively; input the results of N predetermined network layers corresponding to N of the M first sub-models into the second sub-model to obtain the output result of the second sub-model, wherein the output result of the second sub-model includes the feature vector of the target object; wherein M is a positive integer, and N is a positive integer less than or equal to M.

[0017] At least one embodiment of this disclosure provides an electronic device, including an imaging device and an image processing device. The imaging device is configured to capture a detected image, wherein the detected image includes a target object. The image processing device is configured to receive the detected image and perform an image processing method provided in any embodiment of this disclosure based on the detected image.

[0018] At least one embodiment of this disclosure provides an electronic device, including a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the image processing method provided in any embodiment of this disclosure.

[0019] At least one embodiment of this disclosure provides a computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the image processing method provided in any embodiment of this disclosure. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0021] Figure 1A A flowchart of an image processing method provided by at least one embodiment of the present disclosure is shown;

[0022] Figure 1B It shows Figure 1A A flowchart of an example of step S120 shown;

[0023] Figure 2 A schematic diagram of an image detection model provided in at least one embodiment of the present disclosure is shown;

[0024] Figure 3 A schematic diagram of a heat map provided in at least one embodiment of the present disclosure is shown;

[0025] Figure 4A A schematic diagram of a first sub-model provided in at least one embodiment of the present disclosure is shown;

[0026] Figure 4B A schematic diagram of another first sub-model provided by at least one embodiment of the present disclosure is shown;

[0027] Figure 5 A schematic diagram of an initial model provided by at least one embodiment of the present disclosure is shown;

[0028] Figure 6A schematic block diagram of an image processing apparatus provided in at least one embodiment of the present disclosure;

[0029] Figure 7A A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0030] Figure 7B A schematic block diagram of another electronic device provided for at least one embodiment of this disclosure;

[0031] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of this disclosure; and

[0032] Figure 9 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0034] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0035] The inventors discovered that when detecting facial feature vectors from a user's facial image, a two-stage detection method is typically used. This method involves two stages: face detection and face recognition. For example, firstly, the facial image is input into a first model for face detection, resulting in several detection boxes. Then, images within these detection boxes are cropped from the facial image. Finally, these cropped images are fed into a second model for face recognition to extract feature vectors. This approach requires two separate models, and image cropping is necessary between the two models. Therefore, this two-stage detection method is time-consuming and computationally expensive, resulting in slow detection speeds and failing to meet real-time requirements in scenarios such as facial recognition check-in. Furthermore, with the widespread deployment of edge computing, there is a desire for edge devices to support as many models as possible, necessitating the reduction and compression of models.

[0036] At least one embodiment of this disclosure provides an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium. The image processing method includes: acquiring a detection image, the detection image including a target object; inputting the detection image into an image detection model to obtain an output result of the image detection model, the output result of the image detection model including a feature vector of the target object, the feature vector being used to identify the target object; the image detection model including M first sub-models and second sub-models; inputting the detection image into the image detection model to obtain an output result of the image detection model, including: inputting the detection image into each of the M first sub-models; inputting the results of N predetermined network layers corresponding to N of the M first sub-models into the second sub-model to obtain an output result of the second sub-model, the output result of the second sub-model including a feature vector of the target object; wherein M is a positive integer, and N is a positive integer less than or equal to M.

[0037] According to this image processing method, the image to be detected is input into an image detection model. The image detection model can directly obtain the feature vector of the target object in the image to be detected, without needing to crop the image from the original image (i.e., the image to be detected), thus achieving one-stage detection. Therefore, compared with two-stage detection, the image processing method of this disclosure can save time and computing power, accelerate the detection speed, and better meet the requirements of real-time performance.

[0038] It should be noted that, in the embodiments of this disclosure, "one-stage detection" can mean that the detection operation and the recognition operation (e.g., when applied to a face recognition scenario, the detection operation is face detection, such as obtaining a face detection box; the recognition operation is face recognition, such as obtaining a feature vector for face recognition) are implemented using the same model. For example, in a face recognition scenario, a face image can be processed by an image detection model to directly extract the face feature vector, which is then used to achieve face recognition.

[0039] Figure 1A A flowchart of an image processing method provided by at least one embodiment of the present disclosure is shown. Figure 1B It shows Figure 1A A flowchart of an example of step S120 is shown.

[0040] like Figure 1A As shown, the image processing method may include steps S110 to S120.

[0041] Step S110: Obtain the image to be detected, which includes the target object.

[0042] Step S120: Input the image to be detected into the image detection model to obtain the output result of the image detection model. The output result of the image detection model includes the feature vector of the target object, which is used to identify the target object.

[0043] For example, an image detection model includes M first sub-models and second sub-models. Figure 1B As shown, step S120 may include steps S121 to S122.

[0044] Step S121: Input the image to be detected into M first sub-models respectively.

[0045] Step S122: Input the results of the N predetermined network layers corresponding to the N first sub-models in the M first sub-models into the second sub-model to obtain the output result of the second sub-model, wherein the output result of the second sub-model includes the feature vector of the target object.

[0046] For example, M is a positive integer, and N is a positive integer less than or equal to M.

[0047] For example, in step S110, the image to be detected can be acquired through image acquisition, and the target object in the image to be detected can be a face. The face can be a human face; in the embodiments of this disclosure, a human face is used as an example for explanation. It should be noted that in practical applications, the type of target object can be set according to actual needs. Besides a human face, the target object can also be, for example, an animal face or other objects that require feature vector detection.

[0048] For example, in some scenarios (such as a check-in scenario), the number of target objects in the detected image is one, meaning that one detected image contains only one target object. If the detected image contains two or more objects, then one of those objects can be selected as the target object, ignoring the others. For example, the object closest to the center point of the detected image can be selected as the target object, or the object with the largest area can be selected as the target object.

[0049] For example, the image detection model can be a neural network model, which can be pre-trained using samples to obtain a trained image detection model. In step S120, the image to be detected is input into the trained image detection model to obtain the feature vector of the target object.

[0050] For example, a feature vector may contain at least one of the following: contour information, distance information, curvature information, etc. When the target object is a face, the feature vector may describe, for example, the individual features of the face and its multiple facial features (eyes, nose, mouth, etc.) and / or the structural relationships between the face and its multiple facial features. For example, contour information may include overall facial contour information and / or the contour information of facial features, which can be represented by the coordinates of key points on the contour. Distance information may include the distances between multiple facial features and / or the distances between facial features and the overall facial contour. Curvature information may include, for example, the curvature of the overall facial contour and / or the curvature of the facial feature contours.

[0051] Figure 2 A schematic diagram of an image detection model provided by at least one embodiment of the present disclosure is shown.

[0052] like Figure 2 As shown, the image detection model includes, for example, sub-models A1, A2, A3, and B. Each of sub-models A1, A2, and A3 can serve as a first sub-model, and sub-model B can serve as a second sub-model. Each first and second sub-model can be, for example, a neural network model, each including multiple network layers. The image to be detected is input into sub-models A1, A2, and A3 respectively, so that the multiple network layers of each first sub-model are used to sequentially calculate the image. For each first sub-model, after the image is processed by the calculation of each network layer of the first sub-model, the result corresponding to that network layer can be obtained (the result can be a vector, hereinafter referred to as a feature map). The size of the feature map is related to the original size of the image to be detected and the size of the convolution kernel of each network layer. The calculation results (i.e., feature maps) of the predetermined network layers of sub-models A1, A2, and A3 can be input into sub-model B. After processing by multiple network layers of sub-model B, the feature vector of the target object in the image to be detected can be obtained.

[0053] For example, the output of the image detection model also includes the outputs of the M sub-models corresponding to the M first sub-models, where each of the M sub-model outputs corresponds to one of the M types of features. For example, the outputs of the M sub-models may include at least one of heatmap features, bounding box features, and keypoint features.

[0054] For example, each first sub-model outputs at least one type of feature, and the types of features output by each first sub-model are different. For example, as Figure 2 As shown, sub-model A1 can output heatmap features, sub-model A2 can output bounding box features, and sub-model A3 can output keypoint features. For example, after obtaining the output results of one or more first sub-models, the output results of one or more first sub-models can be provided to the user.

[0055] For example, the image to be detected is input into sub-model A1, which can output multiple heatmaps. Each heatmap can show a key point. For example, the position with the highest value in each heatmap represents the position of the key point corresponding to that heatmap. Figure 3 A schematic diagram of a heat map provided in at least one embodiment of the present disclosure is shown. Figure 3 Part (a) represents the image being detected, part (b) represents a heatmap, and part (c) is the superimposed image of parts (a) and (b). For example... Figure 3 As shown, after inputting the detected image in part (a) into sub-model A1, sub-model A1 can output multiple heatmaps, each containing a keypoint. The heatmap shown in part (b) represents, for example, the keypoint at the mouth. Each coordinate position in this heatmap has a corresponding heat value, and different heat values ​​are represented by different color levels or colors. For example, Figure 3 The heat value is highest at the highlighted position P in part (b), and this highlighted position P can be used as a key point in the heat map.

[0056] For example, the image to be detected is input into sub-model A2, which can output one or more detection boxes (e.g., face detection boxes) about the target object. The image to be detected is input into sub-model A3, which can output location information about one or more key points of the target object.

[0057] For example, in some embodiments, after obtaining the output results of one or more first sub-models, the output results of one or more first sub-models can be displayed to the user for reference. In some embodiments, the output results of one or more first sub-models can be overlaid on the detected image and displayed to the user. For example, features such as face detection boxes and key points can be overlaid on the detected image to enrich the visual effects and improve the user experience.

[0058] For example, N can be equal to M, meaning that the results of the predetermined network layers of all M first sub-models can be input into the second sub-model; or, N can be less than M, meaning that the results of the predetermined network layers of some of the M first sub-models can be input into the second sub-model.

[0059] For example, the above embodiments use M=3 as an example for illustration. However, this disclosure is not limited to this. M can be other values, such as M=1, M=2, or M>3. The value of M can be determined according to actual needs.

[0060] For example, the above embodiment uses one second sub-model as an example for illustration. However, in practical applications, two or more second sub-models can be set according to actual needs. That is, the results of the predetermined network layers of N first sub-models can be simultaneously input into two or more second sub-models. Different second sub-models can output different feature vectors, for example.

[0061] For example, when training an image detection model, the various sub-models (e.g., M first sub-models and second sub-models) within the model are trained together. During training, the loss information of the image detection model is calculated jointly based on the outputs of the M first sub-models (e.g., heatmaps, bounding boxes, and keypoints) and the output (feature vector) of the second sub-model. This loss information is then used to simultaneously adjust the parameters of the image detection model (i.e., the parameters of each first sub-model and the parameters of the second sub-model). Through the mutual influence of the M first and second sub-models, the computational results of each network layer in the M first sub-models can include not only the location information of the target object but also the feature information of the target object in the original image. Therefore, it is unnecessary to crop the image from the original image; the feature map containing the feature information of the original image can be directly input into the second sub-model to obtain the feature vector of the target object.

[0062] According to the image processing method of this disclosure, the image to be detected is input into an image detection model. The feature vector of the target object in the image can be directly obtained from the image detection model without having to crop the image from the original image, thus achieving one-stage detection. Therefore, compared with two-stage detection, the image processing method of this disclosure can save time and computing power, accelerate the detection speed, and better meet the requirements of real-time performance.

[0063] Figure 4A A schematic diagram of a first sub-model provided in at least one embodiment of the present disclosure is shown. Figure 4B A schematic diagram of another first sub-model provided by at least one embodiment of the present disclosure is shown.

[0064] like Figure 4A and Figure 4B As shown, for example, each of the M first sub-models may include a first input layer, a plurality of first intermediate layers, and a first output layer connected in sequence. In some embodiments, such as Figure 4A As shown, the multiple first intermediate layers may include multiple depthwise convolutional layers (depthwise convolutional layer 1 to depthwise convolutional layer n, where n is an integer greater than 1) connected in sequence. In other embodiments, such as Figure 4BAs shown, the multiple first intermediate layers may include multiple deep convolutional layers (deep convolutional layer 1 to deep convolutional layer n, where n is an integer greater than 1) and multiple deconvolutional layers (deconvolutional layer 1 to deconvolutional layer m, where m is an integer greater than 1, and m and n may be equal or unequal, for example, m may be less than n).

[0065] For example, depthwise convolutional layers can perform depthwise separable convolution operations.

[0066] For example, Figure 4A The diagram illustrates a network structure based on deep convolutional layers, which is suitable for tasks such as image detection, classification, and feature extraction. Figure 4B The diagram shows a network structure based on deep convolutional layers with added deconvolutional layers. This network structure is suitable for image segmentation, for example. Due to the addition of deconvolution calculation, the feature map output by this network structure can be larger, that is, the output result can be larger.

[0067] For example, for each first sub-model, the size of the output result can be used... Figure 4A and Figure 4B This is one type of network structure shown. For example, sub-model A1 outputs heatmap features. Since the size of the heatmap features is relatively large, sub-model A1 can adopt... Figure 4B The network structure shown is suitable for sub-models with smaller output sizes (e.g., sub-model A2). Figure 4A The network structure shown.

[0068] For example, each first sub-model contains an input layer and an output layer. Since it cannot be guaranteed that the inputs and outputs of all sub-modules are consistent, the form of the input and output layers can be determined according to actual needs, and this disclosure does not impose any restrictions on this. Because each first sub-model contains a complete model structure (input layer, intermediate layer, and output layer), multiple first and second sub-models of the trained image detection model do not affect each other. Therefore, for a trained image detection model, when needed, one or more first or second sub-models of the image detection model can be separated and used as individual network models. For example, in a certain scenario, if it is necessary to obtain detection boxes, sub-model A2 in the image detection model can be separated and used as a separate network model. As another example, if sub-model A3 is not needed when performing feature vector detection, then sub-model A3 can be separated, and sub-models A1, A2, and B can be used to process the detected image to obtain feature vectors. For example, in a scenario where it's necessary to obtain detection box features, heatmap features, and keypoint features, but not feature vectors (i.e., sub-model B is not needed), sub-model B can be separated, retaining sub-models A1, A2, and A3. For instance, the various sub-models within an image detection model can be trained together during model training. After training, one or more sub-models can be separated for individual inference and prediction as needed, without affecting the other sub-models. This approach makes the application of image detection models more flexible and applicable to a wider range of scenarios. Furthermore, the network structure of the sub-models can be arbitrarily designed, and the sub-models do not affect each other, as long as the required output results are met.

[0069] For example, the predetermined network layer corresponding to each first sub-model can be one of multiple first intermediate layers in each first sub-model or the first output layer in each first sub-model. For example, in Figure 4A In the example shown, the predetermined network layer can be a deep convolutional layer or a first output layer; in Figure 4A In the example shown, the predetermined network layer can be a deep convolutional layer, a deconvolutional layer, or a first output layer.

[0070] For example, N predetermined network layers are layers with the same dimension of output from N first sub-models. For example, for each first sub-model, the feature map size obtained by each network layer is different. Layers with the same size of output feature maps from the N first sub-models can be used as predetermined network layers, that is, the dimensions of the output (vector) of these N predetermined network layers are the same.

[0071] For example, step S122 may include: performing calculations on the results of N predetermined network layers to obtain calculation results; and inputting the calculation results into the second sub-model.

[0072] For example, operations include multiplication or addition.

[0073] For example, if the results of N predetermined network layers have the same dimension, multiplication or addition can be performed directly on these N predetermined network layer results. For instance, elements at corresponding positions in the results of these N predetermined network layers can be multiplied or added. For example, if the results of N predetermined network layers are all 1*256 dimensional vectors, and the result of the predetermined network layer of sub-model A1 is, for example, R1 = [p1, p2, ..., p256], and the result of the predetermined network layer of sub-model A2 is, for example, R2 = [q1, q2, ..., q256], then the product of the results of the predetermined network layers of sub-model A1 and sub-model A2 is K = R1 * R2 = [p1 * q1, p2 * q2... p256 * q256]. Then, K can be input into sub-model B. The main consideration for multiplication is to highlight the gradient of features. If two obvious features (with large values) are multiplied, the obvious features will be amplified, while less obvious features will be weakened, thus increasing the accuracy of recognition.

[0074] For example, features from multiple sub-models can be selected for multiplication. For instance, the results of sub-models A1, A2, and A3 can be multiplied, and the product is then input into sub-model B. The specific sub-models whose results are selected for multiplication are determined based on the actual application requirements.

[0075] For example, in some other embodiments, the results of predetermined network layers of N first sub-models may be directly input into the second sub-model without computation.

[0076] For example, the second sub-model includes a second input layer, multiple second intermediate layers, and a second output layer connected in sequence. The multiple second intermediate layers include multiple depthwise convolutional layers connected in sequence; or the multiple second intermediate layers include multiple depthwise convolutional layers and multiple deconvolutional layers connected in sequence. For example, for the second sub-model, the size of the output feature vector can also be determined by... Figure 4A The network structure shown or Figure 4B The network structures shown are as follows. For example, if the size of the output feature vector is greater than the size threshold, the network structure shown in 4B is used; if the size of the output feature vector is not greater than the size threshold, the network structure shown in 4A is used.

[0077] For example, after obtaining the feature vector of the target object, the feature vector of the target object can be compared with multiple feature vectors in the database to determine the target feature vector that satisfies the distance condition with the feature vector of the target object. The label information corresponding to the target feature vector is then used as the recognition result of the target object.

[0078] For example, in a facial recognition check-in scenario, the facial feature vector of each user who needs to check in can be obtained first. The user's label information (e.g., name and / or ID) and its corresponding facial feature vector are stored in a database, with the label information serving as a unique identifier for each user. When a user checks in, an image containing the user's face is captured as the detection image. The facial feature vector in this detection image is obtained using an image detection model. Then, feature vectors that satisfy a distance condition with the feature vector output by the model can be determined from the database as target feature vectors. The label information corresponding to these target feature vectors is used as the recognition result of the target object, thereby identifying the user's name and / or ID.

[0079] For example, the feature vector of the target object and the target feature vector satisfy a distance condition. In some embodiments, the distance (e.g., Euclidean distance) between each feature vector in the database and the feature vector of the target object can be calculated, and then sorted according to the distance from smallest to largest. The feature vector with the smallest distance to the feature vector of the target object in the database can be used as the target feature vector. That is, the distance condition can include the condition of being closest to the feature vector of the target object. In other embodiments, the distance condition can also include the condition that the distance between the feature vector of the target object and the feature vector of the target object is less than a distance threshold. For example, a feature vector can be selected as the target feature vector from one or more feature vectors in the database whose distance to the feature vector of the target object is less than the distance threshold. For example, the recognition result obtained in the above manner can be output to the user, and can be output to the user in the form of, for example, an image or sound.

[0080] For example, as mentioned above, before using an image detection model to predict feature vectors, the image detection model needs to be trained first. The training process is described below.

[0081] For example, multiple sample images and multiple label data corresponding to the multiple sample images can be obtained, where each sample image includes a sample object; based on the multiple sample images and multiple label data, the parameters of the initial model are updated and iterated until the training completion condition is met, and the trained initial model is used as the image detection model.

[0082] For example, a sample object can be a face, and a sample image is an image containing a face. Each sample image has corresponding label data, which can include heatmap label data, bounding box label data, keypoint label data, and feature vector label data. The label data represents the ground truth of the sample, that is, the real data corresponding to the sample image. The label data can be calculated by a predetermined algorithm (such as image processing algorithms such as keypoint detection) or obtained by manual annotation.

[0083] For example, during the process of updating and iterating the parameters of the initial model, the following operations can be performed for each sample image: input the sample image into the initial model to obtain the output of the initial model, wherein the output of the initial model includes the initial feature vector of the sample object in the sample image; calculate the loss information based on the output of the initial model and the label data corresponding to the sample image; and update the parameters of the initial model based on the loss information.

[0084] For example, the initial model includes M first initial sub-models and second initial sub-models. Inputting sample images into the initial model to obtain its output can include: inputting the sample images into each of the M first initial sub-models; inputting the results of the N predetermined network layers corresponding to the N first initial sub-models into the second initial sub-model to obtain its output, which includes the initial feature vectors of the sample objects in the sample images. It should be noted that the definitions of the N predetermined network layers corresponding to the N first initial sub-models can be referenced above, provided there are no contradictions.

[0085] Figure 5 A schematic diagram of an initial model provided by at least one embodiment of this disclosure is shown. For example... Figure 5 As shown, the initial model includes, for example, initial sub-models A1', A2', A3', and B'. Each of these initial sub-models A1', A2', and A3' can serve as a first initial sub-model, and initial sub-model B can serve as a second initial sub-model. Each first and second initial sub-model can be, for example, a neural network model, each including multiple network layers. A sample image (e.g., a first sample image) is input into the initial sub-models A1', A2', and A3' respectively. Then, the results of the predetermined network layers of each of the initial sub-models A1', A2', and A3' can be output to the initial sub-model B'. After processing by the initial sub-model B', the initial feature vector of the sample object in the first sample image can be obtained.

[0086] For example, the output of the initial model also includes the outputs of the M initial sub-models corresponding to the M first initial sub-models, each of which corresponds to one of the M types of features. For example, the outputs of the M initial sub-models may include at least one of initial heatmap features, initial bounding box features, and initial keypoint features. Figure 5 As shown, the initial sub-model A1` outputs the initial heatmap features, the initial sub-model A2` outputs the initial detection box features, and the initial sub-model A3` outputs the initial key point features.

[0087] For example, after obtaining the output results of M initial sub-models and the output results of the second initial sub-model, the loss function can be used to calculate the loss information (e.g., the total loss value) by processing the output results of the M initial sub-models, the output results of the second initial sub-model, and the label data.

[0088] For example, after obtaining the initial output results (initial feature vector, initial heatmap features, initial detection box features, and initial keypoint features) corresponding to the first sample image using the initial model, a loss function can be used to calculate the initial output results and label data of the first sample image to obtain the first loss information. Based on this first loss information, the parameters of the M first initial sub-models and the second initial sub-model are adjusted and updated to obtain the updated initial model. Then, the next sample image (e.g., the second sample image) is input into the updated initial model to obtain the initial output results corresponding to the second sample image. The loss function is then used to calculate the initial output results and label data of the second sample image to obtain the second loss information. This second loss information is then used to adjust and update the parameters (the parameters after the first update) of the M first initial sub-models and the second initial sub-model a second time to obtain the updated initial model. This process is repeated, using multiple sample images to continuously update and iterate the parameters of the initial model until the training completion condition is met, at which point training can end. The trained initial model (i.e., the initial model after the last update) is used as the image detection model. Training completion conditions include, for example, the loss information being less than the loss threshold, or the training time exceeding the time threshold.

[0089] The initial sub-model A1` after training is... Figure 2 The image detection model shown has a sub-model A1, and the initial sub-model A2' after training is... Figure 2 The image detection model shown has a sub-model A2, and the initial sub-model A3' after training is... Figure 2 The sub-model A3 in the image detection model shown is the initial sub-model B' after training. Figure 2 Sub-model B in the image detection model shown.

[0090] It should be noted that the parameters of the initial model can also be adjusted multiple times using the same sample image.

[0091] For example, the loss function consists of two parts: a first sub-function concerning the second initial sub-model and a second sub-function concerning the M first initial sub-models.

[0092] For example, the first sub-function can be used in conjunction with softmax loss and center loss. The first sub-function can be expressed as:

[0093]

[0094] In equation (1) above, L is the loss value of the first sub-function. S L represents the loss value corresponding to softmax loss. C Let y represent the loss value corresponding to center loss, and let y represent the model output (i.e., the initial feature vector output by the second initial sub-model). i This represents the i-th row (i.e., the y-th row) of the model output. i (categories), The y-th element of the model output represents the... i The feature centers of each category, b represents the bias, and x... i Indicates belonging to the y-th i The depth features of the i-th layer of the category, b j b represents the offset of the j-th column of the network. yi Indicates the yth i Category offset, This indicates that the input is x. i The offset is b yi Network weight at time, λ represents the weight of column j, m represents the batch size, n represents the number of categories, and λ represents the weight coefficient, which is user-defined and is usually 0.5.

[0095] For example, the second sub-function may include: the loss from the heatmap, the loss from predicting the length and width of the target object, and the loss from the center point offset of the target object. The second sub-function can be expressed as:

[0096] L det =L k +λ size L size +λ off L off (2)

[0097] In equation (2), L det L represents the loss value of the second subfunction. k L represents the loss value of the heatmap. size L represents the predicted loss value for length and width. off The loss value, λ, represents the center point offset. size and λ off λ is a hyperparameter. size For example, λ is 0.1. off For example, it could be 1.

[0098] For example, L k L can be calculated using heatmap label data and the initial heatmap features output by the initial sub-model A1'.k It can be represented as:

[0099]

[0100] In equation (3), α and β are hyperparameters, α is, for example, 2, β is, for example, 4, N represents the number of key points in the heatmap, the subscript xyc of the summation symbol ∑ represents all coordinate points on all heatmaps, c represents the category, and each category corresponds to one heatmap. Y represents the model's predicted values ​​(i.e., initial heatmap features). xyc This represents the actual value in the label data (i.e., the heatmap label data).

[0101] For example, L size and L off L can be calculated using the detection box label data and the initial detection box features output by the initial sub-model A2'. size It can be represented as:

[0102]

[0103] In equation (4), N represents the number of detection boxes, and k represents the k-th detection box. This represents the predicted size of the detection box (i.e., the size data in the initial detection box features), s k This represents the actual size (i.e., the size data in the detection box label data).

[0104] For example, L off It can be represented as:

[0105]

[0106] In equation (5), N represents the number of detection boxes, and p represents the true coordinates of the center point of the detection box (i.e., the center point offset value in the initial detection box features). R represents the predicted value of the center point offset (i.e., the center point offset data in the initial detection box features), and R represents the scaling scale. Represents the approximate integer coordinates of the center point after scaling.

[0107] In addition, the second sub-function can also include the loss L at the key points. lm For example, λ can be added to equation (2) above. lm L lm Therefore, the second sub-function can be expressed as:

[0108] L det =L k +λ size L size +λ off L off +λ lmL lm (6)

[0109] λ lm For hyperparameters, L lm L can be calculated using keypoint label data and the initial keypoint features output by the initial sub-model A3'. lm It can be represented as:

[0110]

[0111] In equation (7), The loss value representing the x-coordinate of the keypoint. The loss value representing the y-coordinate of the keypoint, lm x The lm represents the predicted x-coordinate of the key point. y c represents the predicted y-coordinate of the key point. x c represents the true x-coordinate of the key point. y The box represents the true y-coordinate of the key point. w This indicates the width of the box. h Indicates the height of the box.

[0112] For example, the total loss value L total L is the sum of the loss values ​​of the first sub-function and the second sub-function. total =L+L det For example, the total loss value L total It can also be a weighted sum of the loss values ​​of the first sub-function and the second sub-function, i.e., L. total =a1*L+a2*L det a1 represents the weight of the first sub-function L, and a2 represents the weight of the second sub-function. The specific values ​​of a1 and a2 can be set according to the actual situation.

[0113] For example, the first sub-function can assist in the extraction of feature vectors, helping to reduce the intra-class gap between feature vectors of faces with the same identity and increase the distance between feature vectors of faces with different identities. The second sub-function can help the detection model network and improve the accuracy of face detection boxes, etc. Therefore, by combining these two parts to train the model, the advantages of both parts can be combined, so that the model can have the effects brought by the above two sub-functions at the same time.

[0114] For example, in some embodiments, the results of the N predetermined network layers corresponding to the N first initial sub-models can be processed to obtain the calculation result, and then the calculation result can be input into the second initial sub-model. The calculation includes multiplication or addition.

[0115] For example, the N predetermined network layers are layers with the same dimension in the N first initial sub-models, that is, the dimensions of the results (e.g., vectors) of the N predetermined network layers are the same.

[0116] For example, each of the M first initial sub-models includes a first initial input layer, multiple first initial intermediate layers, and a first initial output layer connected in sequence. The multiple first initial intermediate layers include multiple initial depthwise convolutional layers connected in sequence; or the multiple first initial intermediate layers include multiple initial depthwise convolutional layers and multiple initial deconvolutional layers connected in sequence.

[0117] For example, the predetermined network layer corresponding to each of the N first initial sub-models is one of the multiple first initial intermediate layers in each first initial sub-model or the first initial output layer in each first initial sub-model.

[0118] For example, the second initial sub-model includes a second initial input layer, multiple second initial intermediate layers, and a second initial output layer connected in sequence. The multiple second initial intermediate layers include multiple initial depthwise convolutional layers connected in sequence; or the multiple second initial intermediate layers include multiple initial depthwise convolutional layers and multiple initial deconvolutional layers connected in sequence.

[0119] For example, an image detection model is obtained by adjusting the parameters of an initial model. During training, the network structure and data processing flow of the model do not need to be adjusted. Therefore, the network structure and data processing flow of the image detection model are, for example, consistent with the network structure and data processing flow of the initial model.

[0120] At least one embodiment of this disclosure also provides an image processing apparatus. Figure 6 This is a schematic block diagram of an image processing apparatus provided for at least one embodiment of the present disclosure.

[0121] like Figure 6 As shown, the image processing apparatus 200 may include an acquisition module 201 and a detection module 202. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, these modules can be implemented as hardware (e.g., circuit) modules, software modules, or any combination of both, as is the case in the following embodiments, and will not be repeated here. For example, these units can be implemented using a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, along with corresponding computer instructions. It should be noted that... Figure 6 The components and structure of the image processing apparatus 200 shown are merely exemplary and not limiting. The image processing apparatus 200 may also have other components and structures as needed.

[0122] For example, the acquisition module 201 is configured to acquire a detected image, which includes a target object.

[0123] For example, the detection module 202 is configured to input the image to be detected into an image detection model and obtain the output result of the image detection model, wherein the output result of the image detection model includes the feature vector of the target object, and the feature vector is used to identify the target object.

[0124] For example, the image detection model includes M first sub-models and second sub-models. The detection module 202 is further configured to: input the image to be detected into the M first sub-models respectively; input the results of the N predetermined network layers corresponding to the N first sub-models in the M first sub-models into the second sub-model to obtain the output result of the second sub-model, the output result of the second sub-model including the feature vector of the target object; M is a positive integer, and N is a positive integer less than or equal to M.

[0125] For example, the acquisition module 201 and the detection module 202 may include code and programs stored in memory; the processor may execute the code and programs to implement some or all of the functions of the acquisition module 201 and the detection module 202 as described above. For example, the acquisition module 201 and the detection module 202 may be dedicated hardware devices used to implement some or all of the functions of the acquisition module 201 and the detection module 202 as described above. For example, the acquisition module 201 and the detection module 202 may be a circuit board or a combination of multiple circuit boards used to implement the functions described above. In the embodiments of this application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) processor-executable firmware stored in memory.

[0126] It should be noted that the acquisition module 201 can be used to implement Figure 1A As shown in step S110, the detection module 202 can be used to implement... Figure 1A and 1B The step S120 is shown. Therefore, for a detailed description of the functions that the acquisition module 201 and the detection module 202 can achieve, please refer to the relevant descriptions of steps S110 to S120 in the embodiments of the above image processing method; repeated details will not be repeated here. Furthermore, the image processing apparatus 200 can achieve similar technical effects to the aforementioned image processing method, which will not be described further here.

[0127] It should be noted that, in the embodiments of this disclosure, the image processing apparatus 200 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0128] For example, in some embodiments, the image processing apparatus 200 may further include a recognition module configured to: compare the feature vector of the target object with multiple feature vectors in a database to determine a target feature vector that satisfies the distance condition with the feature vector of the target object from the multiple feature vectors, and use the label information corresponding to the target feature vector as the recognition result of the target object.

[0129] For example, in some embodiments, the image processing apparatus 200 may further include a training module configured to: acquire multiple sample images and multiple label data corresponding to the multiple sample images, each sample image including a sample object; update and iterate the parameters of an initial model based on the multiple sample images and the multiple label data until the training completion condition is met, and use the trained initial model as an image detection model.

[0130] This disclosure also provides an electronic device in some embodiments. Figure 7A This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0131] For example, such as Figure 7A As shown, the electronic device 300 includes a shooting device 301 and an image processing device 302.

[0132] For example, the imaging device 301 is configured to capture an image to be detected, the image including the target object.

[0133] For example, the image processing apparatus 302 is configured to receive a detected image and perform the image processing method as described in any of the above embodiments based on the detected image.

[0134] For example, the shooting device 301 can be a rear camera of an electronic device, or a front camera and reflective device of an electronic device.

[0135] For example, the image processing device 302 can be implemented as a central processing unit, a dedicated processing chip, a digital signal processor, etc., and this disclosure does not impose any specific limitations on it.

[0136] For example, electronic device 300 can be a terminal device such as a time clock, and can also provide a display unit (such as a touch screen). For example, the display unit can provide a corresponding human-computer interaction interface to display the response of the interactive operation, interactive action prompts, etc. This disclosure does not impose specific limitations in this regard.

[0137] For example, a detailed description of the process by which the electronic device 300 performs the image processing method can be found in the relevant description in the embodiments of the above-described image processing method, and repeated descriptions will not be repeated here.

[0138] Some embodiments of this disclosure also provide another electronic device. Figure 7B This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0139] For example, such as Figure 7B As shown, the electronic device 400 includes a processor 401 and a memory 402. It should be noted that... Figure 7B The components of the electronic device 400 shown are merely exemplary and not limiting. The electronic device 400 may have other components as needed for the actual application.

[0140] For example, processor 401 and memory 402 can communicate with each other directly or indirectly.

[0141] For example, processor 401 and memory 402 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. Processor 401 and memory 402 can also communicate with each other via a system bus, and this disclosure is not limiting in this regard.

[0142] For example, in some embodiments, memory 402 is used to store computer-readable instructions non-transitory. When processor 401 executes the computer-readable instructions, the computer-readable instructions are executed by processor 401 to implement the image processing method according to any of the above embodiments. For specific implementations and related explanations of the various steps of this image processing method, please refer to the embodiments of the image processing method described above; repeated details will not be elaborated here.

[0143] For example, processor 401 and memory 402 can be located on the server side (or in the cloud).

[0144] For example, processor 401 can control other components in electronic device 400 to perform desired functions. Processor 401 can be a central processing unit (CPU), graphics processing unit (GPU), network processor (NP), etc.; it can also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be based on x86 or ARM architectures, etc.

[0145] For example, memory 402 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and processor 401 may execute the computer-readable instructions to implement various functions of electronic device 400. Various application programs and various data may also be stored in the storage medium.

[0146] For example, in some embodiments, the electronic device 400 can be a mobile phone, tablet computer, electronic paper, television, monitor, laptop computer, digital photo frame, navigator, wearable electronic device, smart home device, etc.

[0147] For example, electronic device 400 may include a display panel, which can be used for image segmentation, etc. For example, the display panel can be a rectangular panel, a circular panel, an elliptical panel, or a polygonal panel. Furthermore, the display panel can be not only a flat panel, but also a curved panel, or even a spherical panel.

[0148] For example, electronic device 400 can have touch functionality, that is, electronic device 400 can be a touch device.

[0149] For example, a detailed description of the process by which the electronic device 400 performs the image processing method can be found in the relevant description in the embodiments of the above-described image processing method, and repeated descriptions will not be repeated here.

[0150] Figure 8 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as Figure 8As shown, computer-readable instructions 501 are non-transitory stored on the computer-readable storage medium 500. For example, when the computer-readable instructions 501 are executed by a processor, one or more steps in the image processing method described above can be performed.

[0151] For example, the storage medium 500 can be used in the aforementioned electronic device 400. For example, the storage medium 500 may include the memory 402 in the electronic device 400.

[0152] For example, the description of storage medium 500 can be found in the description of memory 402 in the embodiment of electronic device 400, and the repeated parts will not be repeated.

[0153] Figure 9 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. The electronic device provided in this disclosure can be applied to an Internet system.

[0154] use Figure 9 The computer system provided herein can implement the functions of the image processing apparatus and / or electronic device involved in this disclosure. Such computer systems may include personal computers, laptops, tablets, mobile phones, personal digital assistants, smart glasses, smartwatches, smart rings, smart helmets, and any smart portable or wearable device. A specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. This computer device can be a general-purpose computer device or a purpose-specific computer device. Both types of computer devices can be used to implement the image processing apparatus and / or electronic device of this embodiment. The computer system may include any components necessary to implement the image processing described herein. For example, the computer system can be implemented by a computer device through its hardware, software programs, firmware, and combinations thereof. For convenience, Figure 9 Although only one computer device is shown in the figure, the computer functions related to the information required for image processing described in this embodiment can be implemented in a distributed manner by a set of similar platforms, thus distributing the processing load of the computer system.

[0155] like Figure 9As shown, the computer system may include a communication port 650, connected to a network for data communication. For example, the computer system can send and receive information and data through the communication port 650, enabling wireless or wired communication between the computer system and other electronic devices to exchange data. The computer system may also include a processor group 620 (i.e., the processor described above) for executing program instructions. The processor group 620 may consist of at least one processor (e.g., a CPU). The computer system may include an internal communication bus 610. The computer system may include different forms of program storage units and data storage units (i.e., the memory or storage media described above), such as a hard disk 670, read-only memory (ROM) 630, and random access memory (RAM) 640, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor group 620. The computer system may also include an input / output component 660 for implementing input / output data flow between the computer system and other components (e.g., user interface 680, etc.).

[0156] Typically, the following devices can be connected to the input / output component 660: input devices such as touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices such as displays (e.g., LCD, OLED displays, etc.), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication interfaces.

[0157] Although Figure 9 A computer system with various devices is shown, but it should be understood that the computer system is not required to have all the devices shown, and alternatively, the computer system may have more or fewer devices.

[0158] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0159] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0160] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0161] The following points should be noted regarding this disclosure:

[0162] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0163] (2) For clarity, the thickness and dimensions of layers or structures are enlarged in the accompanying drawings used to describe embodiments of the invention. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element, or there may be intermediate elements present.

[0164] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0165] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. An image processing method, comprising: Acquire a detection image, wherein the detection image includes a target object; The image to be detected is input into an image detection model to obtain the output result of the image detection model, wherein the output result of the image detection model includes the feature vector of the target object, and the feature vector is used to identify the target object; The image detection model includes M first sub-models and second sub-models; The image to be detected is input into the image detection model to obtain the output result of the image detection model, including: The detected image is input into the M first sub-models respectively; The results of the N predetermined network layers corresponding to the N first sub-models in the M first sub-models are input into the second sub-model to obtain the output result of the second sub-model, wherein the output result of the second sub-model includes the feature vector of the target object; Where M is a positive integer, and N is a positive integer less than or equal to M. The results of the N predetermined network layers, including the location and feature information of the target object, are used to replace the image obtained by cropping the detected image and are input into the second sub-model. The output of the image detection model also includes the outputs of the M sub-models corresponding to the M first sub-models, wherein the outputs of the M sub-models correspond to M types of features.

2. The image processing method according to claim 1, wherein, The target object is the face.

3. The image processing method according to claim 1 or 2, further comprising: The feature vector of the target object is compared with multiple feature vectors in the database to determine the target feature vector that satisfies the distance condition with the feature vector of the target object. The label information corresponding to the target feature vector is used as the recognition result of the target object.

4. The image processing method according to claim 1 or 2, wherein, Each of the M first sub-models includes a first input layer, multiple first intermediate layers and a first output layer connected in sequence, and the predetermined network layer corresponding to each of the N first sub-models is one of the multiple first intermediate layers in each first sub-model or the first output layer in each first sub-model. The second sub-model includes a second input layer, multiple second intermediate layers, and a second output layer connected in sequence.

5. The image processing method according to claim 4, wherein, The N predetermined network layers are layers of the results with the same dimension in the N first sub-models.

6. The image processing method according to claim 4, wherein, The plurality of first intermediate layers includes a plurality of depthwise convolutional layers connected in sequence; or the plurality of first intermediate layers includes a plurality of depthwise convolutional layers and a plurality of deconvolutional layers connected in sequence. The plurality of second intermediate layers include a plurality of depth convolutional layers connected in sequence; Alternatively, the plurality of second intermediate layers may include a plurality of depthwise convolutional layers and a plurality of deconvolutional layers connected in sequence.

7. The image processing method according to claim 1 or 2, wherein, The results of the N predetermined network layers corresponding to the N first sub-models in the M first sub-models are input into the second sub-model, including: The results of the N predetermined network layers are processed to obtain the calculation result; The calculation result is input into the second sub-model.

8. The image processing method according to claim 7, wherein, The operations include multiplication or addition. The results of the N predetermined network layers have the same dimension.

9. The image processing method according to claim 1, wherein, The output of the M sub-models includes at least one of heatmap features, detection box features, and key point features.

10. The image processing method according to claim 1 or 2, further comprising: Acquire multiple sample images and multiple label data corresponding to the multiple sample images, wherein each sample image includes a sample object; Based on the multiple sample images and the multiple label data, the parameters of the initial model are updated and iterated until the training completion condition is met, and the trained initial model is used as the image detection model.

11. The image processing method according to claim 10, wherein, Based on the multiple sample images and the multiple label data, the parameters of the initial model are updated and iterated, including performing the following operations for each sample image: The sample image is input into the initial model to obtain the output result of the initial model, wherein the output result of the initial model includes the initial feature vector of the sample object in the sample image; Based on the output of the initial model and the label data corresponding to the sample images, the loss information is calculated. Based on the loss information, update the parameters of the initial model; The initial model includes M first initial sub-models and second initial sub-models; The sample image is input into the initial model to obtain the output of the initial model, including: The sample images are respectively input into the M first initial sub-models; The results of the N predetermined network layers corresponding to the N first initial sub-models in the M first initial sub-models are input into the second initial sub-model to obtain the output result of the second initial sub-model. The output result of the second initial sub-model includes the initial feature vector of the sample object in the sample image.

12. The image processing method according to claim 11, wherein, The output of the initial model also includes the output of the M initial sub-models corresponding to the M first initial sub-models; Based on the output of the initial model and the label data corresponding to the sample images, loss information is calculated, including: The loss information is obtained by using the loss function to calculate the output results of the M initial sub-models, the output result of the second initial sub-model, and the label data.

13. An image processing apparatus, comprising: The acquisition module is configured to acquire a detected image, wherein the detected image includes a target object; The detection module is configured to input the image to be detected into an image detection model and obtain the output result of the image detection model, wherein the output result of the image detection model includes the feature vector of the target object, and the feature vector is used to identify the target object; The image detection model includes M first sub-models and second sub-models; The detection module is further configured as follows: The detected image is input into the M first sub-models respectively; The results of the N predetermined network layers corresponding to the N first sub-models in the M first sub-models are input into the second sub-model to obtain the output result of the second sub-model, wherein the output result of the second sub-model includes the feature vector of the target object; Where M is a positive integer, and N is a positive integer less than or equal to M. The results of the N predetermined network layers, including the location and feature information of the target object, are used to replace the image obtained by cropping the detected image and are input into the second sub-model. The output of the image detection model also includes the outputs of the M sub-models corresponding to the M first sub-models, wherein the outputs of the M sub-models correspond to M types of features.

14. An electronic device comprising: An imaging device configured to capture an image to be detected, wherein the image to be detected includes a target object; An image processing apparatus configured to receive the detected image and perform the image processing method according to any one of claims 1-12 based on the detected image.

15. An electronic device comprising: processor; Memory, including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the image processing method according to any one of claims 1-12.

16. A computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the image processing method according to any one of claims 1-12.

Images