Image element approximation method and terminal device

Abstract

The present application discloses an image element approximation method and terminal device. The method identifies each element in the image to be processed using an image element recognition model based on an artificial neural network, obtaining a recognition result and a category probability table for each element. The recognition result is the candidate category with the highest predicted probability in the category probability table. The image element approximation method disclosed in the present application excludes the candidate category related to the recognition result in the category probability table, and then reselects the candidate category with the highest predicted probability in the category probability table as the approximate result output. The image element approximation method disclosed in the present application abandons the traditional image recognition model's pursuit of accuracy and selects the approximate recognition result of the image element for output, which can effectively inspire the user's imagination and open up the user's creative inspiration.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image element approximation method and terminal device. Background Art

[0002] Image element recognition has long been a hot topic among scientists both domestically and internationally. This research primarily involves identifying the specific location and classification of target elements. Image element recognition is closely intertwined with our daily lives and is a crucial research area in computer vision. With the rapid development of industries such as artificial intelligence, the Internet of Things, and big data, image element recognition is widely used in target tracking, video surveillance, autonomous driving, medical image analysis, and drone navigation.

[0003] However, existing image element recognition technologies often pursue the accuracy of element recognition, and rarely make analogies based on the similarity between elements to display approximate recognition results of image elements to inspire users' imagination. Summary of the Invention

[0004] The purpose of this application is to provide an image element approximation method and terminal device, which can improve the above-mentioned problems.

[0005] The embodiment of the present application is implemented as follows:

[0006] In a first aspect, the present application provides an image element approximation method, comprising:

[0007] S11, obtaining an image to be processed;

[0008] S12. Identify each element in the image to be processed using an image element recognition model to obtain a recognition result and a category probability table for each element, wherein the category probability table includes the probability of the element belonging to each candidate category;

[0009] S13, removing the candidate categories related to the recognition result from the category probability table, and updating the category probability table;

[0010] S14: Use the candidate category with the largest probability in the updated category probability table as the first approximate result of the corresponding element.

[0011] It can be understood that the present application discloses an image element approximation method, which identifies each element in the image to be processed through an image element recognition model based on an artificial neural network, and obtains a recognition result and a category probability table for each element; wherein the recognition result is the alternative category with the highest predicted probability in the category probability table. The image element approximation method disclosed in the present application excludes the alternative categories related to the recognition result in the category probability table, and then reselects the alternative category with the highest predicted probability in the category probability table as the approximate result output. The image element approximation method disclosed in the present application abandons the traditional image recognition model's pursuit of accuracy and selects the approximate recognition results of image elements for output, which can well inspire the user's imagination and open up the user's creative inspiration.

[0012] In an optional embodiment of the present application, the image element recognition model includes an input layer, a composite hidden layer, a prediction layer and an output layer; the input layer is used to obtain an input image; the composite hidden layer includes at least two convolution-pooling modules arranged in sequence, for extracting a feature image of the input image; the prediction layer includes two fully connected layers, for predicting the prediction box position information, the prediction box credibility score and the category probability table of each element in the input image based on the feature image; the output layer is used to output the prediction result of the prediction layer.

[0013] The prediction layer works as follows: the feature image is divided into S×S grids, with each grid corresponding to B prediction boxes. The prediction box with a prediction box confidence score higher than the threshold is used as the target prediction box, which is assumed to contain the image element object to be identified. The image element in the target prediction box is predicted to have a probability of belonging to each candidate category. The prediction box confidence score is used to represent the probability of the object to be identified within the prediction box. A higher prediction box confidence score indicates a greater probability of the object to be identified within the prediction box, meaning that the prediction box contains an object. A lower prediction box confidence score indicates a lower probability of the object to be identified within the prediction box, meaning that the prediction box does not contain an object.

[0014] In a second aspect, the present application provides another image element approximation method, which includes:

[0015] S21. Training the image element recognition model using a locally stored facial image set, where the facial image set includes images of each facial region of each image in a local target album;

[0016] S22, obtaining an image to be processed;

[0017] S23. Recognize each facial region in the image to be processed using an image element recognition model to obtain a recognition result for each facial region and a candidate person probability table, wherein the candidate person probability table includes a probability that the facial region belongs to each candidate person;

[0018] S24, removing the candidate corresponding to the recognition result from the candidate probability table, and updating the candidate probability table;

[0019] S25: Use the candidate with the highest probability in the updated candidate probability table as the first approximate result corresponding to the face area.

[0020] It will be understood that the image element approximation method disclosed in the second aspect of this application trains the image element recognition model using a locally stored set of facial images, and then, according to the approximation method disclosed in the first aspect, uses the trained image element recognition model to output approximate results for each facial region in the image to be processed. After being trained on the locally stored set of facial images, the image element recognition model is capable of recognizing each locally stored portrait. When performing approximate recognition of a target facial region in the image to be processed, in addition to the true result, the recognition result of the person closest to the target facial region among the locally stored persons can be obtained, thereby enhancing the fun of facial recognition.

[0021] In a third aspect, the present application provides another image element approximation method, which includes:

[0022] S31, training the image element recognition model using a target image set, where the target image set includes a plurality of sample images that have been classified in at least two levels;

[0023] S32, obtaining an image to be processed;

[0024] S33. Identify each element in the image to be processed using the trained image element recognition model to obtain a recognition result and a category probability table for each element, wherein the category probability table includes a probability that the element belongs to each candidate category, where the candidate category is the final category of the at least two-level classification.

[0025] S34, removing candidate categories from the category probability table that belong to the same upper-level category as the recognition result, and updating the category probability table;

[0026] S35, taking the candidate category with the largest probability in the updated category probability table as the first approximate result of the corresponding element;

[0027] S36, obtaining an image element approximation model with the same network structure as the image element recognition model;

[0028] S37, removing sample images from the target image set that belong to the same upper-level category as the recognition result, and updating the target image set;

[0029] S38, training the image element approximation model using the updated target image set;

[0030] S39. Recognize each element in the image to be processed by using the trained image element approximation model, and use the recognition result obtained by the image element approximation model as the second approximation result of each element.

[0031] It can be understood that the third aspect of the present application discloses an image element approximation method. On the one hand, after the image element recognition model trained with the target image set recognizes each element in the image to be processed, the alternative categories in the category probability table that belong to the same upper-level classification as the recognition result are removed, and the alternative category with the largest probability is selected from the remaining category probability table as the first approximate result of the corresponding element. On the other hand, after removing the sample images that belong to the same upper-level classification as the above-mentioned recognition result from the target image set, the image element approximation model is trained with the remaining target image set. Then, the image element approximation model is naturally unable to recognize the true recognition result of the image element, and can only select the alternative category with the largest probability from other alternative categories as the second approximate result of the corresponding element. The above method obtains two approximate results by means of the image element recognition model and the image element approximation model, which can better inspire the user's imagination and open up the user's creative inspiration.

[0032] In an optional embodiment of the present application, the image element recognition model is trained using a target image set according to the following loss function:

[0033] LOSS=L 11 +L 12 +L2+L3;

[0034] in,

[0035]

[0036] Among them, λ and λ noobj is the weight corresponding to the loss factor, which is generally a preset constant; Indicates that if the j-th prediction box corresponding to the i-th grid of the feature image of the input image of the image element recognition model is responsible for predicting the image element, then the value is assigned to 1, otherwise it is assigned to 0; S 2 Represents the total number of grids in the input image, and B represents the total number of prediction boxes corresponding to each grid; (x i ,y i ) represents the real box coordinates of the image element corresponding to the i-th grid, Indicates the predicted box coordinates corresponding to the i-th grid, (w i ,h i ) represents the width and height of the ground truth box of the image element corresponding to the i-th grid, Represents the width and height of the prediction box corresponding to the i-th grid; C i represents the credibility score of the real box corresponding to the i-th grid, Represents the credibility score of the prediction box corresponding to the i-th grid; Indicates that if the j-th prediction box corresponding to the i-th grid of the feature image of the input image of the image element recognition model is not responsible for predicting the image element, then the value is assigned to 1, otherwise it is assigned to 0; Indicates that if the prediction box corresponding to the i-th grid of the feature image of the input picture of the image element recognition model contains the image element, it is assigned a value of 1, otherwise it is assigned a value of 0; i (c) represents the probability of predicting the real box element corresponding to the i-th grid of the input image as category c, It represents the probability of predicting the predicted box element corresponding to the i-th grid of the input image as category c, and classes represents the total set of category c.

[0037] In the embodiment of the present application, the network structure and working principle of the image element approximation model and the image element recognition model are the same, and will not be repeated here.

[0038] In a fourth aspect, the present application discloses a terminal device comprising a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method as described in any one of the first to third aspects.

[0039] In a fifth aspect, the present application discloses a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described in any one of the first to third aspects.

[0040] Beneficial effects:

[0041] This application discloses an image element approximation method. This method uses an artificial neural network-based image element recognition model to identify each element in the image to be processed, obtaining a recognition result and a category probability table for each element. The recognition result is the candidate category with the highest predicted probability in the category probability table. The image element approximation method disclosed in this application eliminates the candidate categories associated with the recognition result in the category probability table and then reselects the candidate category with the highest predicted probability in the category probability table as the approximate result for output. The image element approximation method disclosed in this application abandons the traditional image recognition model's pursuit of accuracy and instead selects approximate recognition results for image elements for output, which can effectively inspire users' imagination and broaden their creative inspiration.

[0042] The second aspect of this application discloses an image element approximation method. The image element recognition model is trained using a locally stored set of facial images. The trained image element recognition model then outputs approximate results for each facial region in the image being processed, using the approximation method disclosed in the first aspect. After being trained using the locally stored set of facial images, the image element recognition model is capable of recognizing each locally stored portrait. When approximating a target facial region in the image being processed, the recognition result for the person closest to the target facial region among the locally stored persons is obtained, thereby enhancing the fun of facial recognition.

[0043] The third aspect of the present application discloses an image element approximation method. On the one hand, after the image element recognition model trained on the target image set recognizes each element in the image to be processed, the alternative categories in the category probability table that belong to the same upper-level classification as the recognition result are removed, and the alternative category with the largest probability is selected from the remaining category probability table as the first approximate result of the corresponding element. On the other hand, after removing the sample images that belong to the same upper-level classification as the above-mentioned recognition result from the target image set, the image element approximation model is trained with the remaining target image set. Then, the image element approximation model is naturally unable to recognize the true recognition result of the image element, and can only select the alternative category with the largest probability from other alternative categories as the second approximate result of the corresponding element. The above method obtains two approximate results by means of the image element recognition model and the image element approximation model, which can better inspire the user's imagination and open up the user's creative inspiration.

[0044] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, optional embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is a schematic diagram of the network structure of the image element recognition model provided by this application;

[0047] Figure 2 This is a schematic diagram of the output results of the image element recognition model provided by this application;

[0048] Figure 3 This is a diagram of an interface for simultaneously outputting the first approximate result and the second approximate result of image elements in an image to be processed provided by the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] In a first aspect, the present application provides an image element approximation method, comprising:

[0051] S11. Obtain an image to be processed.

[0052] The image element approximation method provided in this application is applied to smart terminals such as mobile phones and tablets. Users can use the camera application to capture the image to be processed or download the image to be processed from the Internet. Any element that constitutes an object or person in the image to be processed is an image element to be recognized.

[0053] S12. Identify each element in the image to be processed using the image element recognition model to obtain a recognition result and a category probability table for each element. The category probability table includes the probability that the element belongs to each candidate category.

[0054] In an optional embodiment of the present application, the image element recognition model belongs to an artificial neural network model, including an input layer, a composite hidden layer, a prediction layer and an output layer; the input layer is used to obtain an input image; the composite hidden layer includes at least two convolution-pooling modules arranged in sequence, which are used to extract the feature image of the input image; the prediction layer includes two fully connected layers, which are used to predict the prediction box position information, prediction box credibility score and category probability table of each element in the input image based on the feature image; the output layer is used to output the prediction result of the prediction layer.

[0055] like Figure 1 The image element recognition model shown includes an input layer 101, a composite hidden layer 102, a prediction layer 103, and an output layer 104. After the input layer 101 obtains the input image 10, the m convolution-pooling modules of the composite hidden layer 102 repeatedly perform convolution and pooling operations on the input image 10 to extract the feature image of the input image 10; the two fully connected layers in the prediction layer 103 predict the prediction box position information, prediction box confidence score, and category probability table of each element in the input image 10 based on the feature image. The output layer 104 outputs the prediction result based on the prediction box position information, prediction box confidence score, and category probability table. Figure 1 As can be seen from the figure, the input image 10 includes two target elements, whose categories are c1 and c2 respectively.

[0056] The prediction layer works as follows: the feature image is divided into S×S grids, with each grid corresponding to B prediction boxes. The prediction box with a prediction box confidence score higher than the threshold is used as the target prediction box, which is assumed to contain the image element object to be identified. The image element in the target prediction box is predicted to have a probability of belonging to each candidate category. The prediction box confidence score is used to represent the probability of the object to be identified within the prediction box. A higher prediction box confidence score indicates a greater probability of the object to be identified within the prediction box, meaning that the prediction box contains an object. A lower prediction box confidence score indicates a lower probability of the object to be identified within the prediction box, meaning that the prediction box does not contain an object.

[0057] like Figure 2As shown, the prediction layer predicts the prediction box position information, prediction box confidence score, and category probability table for each element in the input image 10 based on the feature image 100. In the figure, the current grid of the feature image 100 of the input image 10 corresponds to two prediction boxes b1 and b2. (x1, y1, h1, w1) are the prediction box position information of prediction box b1, that is, the horizontal coordinate, vertical coordinate, height, and width of prediction box b1, and C1 is the prediction box confidence score of prediction box b1; (x2, y2, h2, w2) are the prediction box position information of prediction box b2, that is, the horizontal coordinate, vertical coordinate, height, and width of prediction box b2, and C2 is the prediction box confidence score of prediction box b2; p(c1) represents the predicted probability that the element in the prediction box belongs to category c1, and p(c2) represents the predicted probability that the element in the prediction box belongs to category c2. And so on, a total of m alternative category probabilities are predicted.

[0058] S13. Remove the candidate categories related to the recognition results in the category probability table and update the category probability table.

[0059] It can be understood that the recognition result is the candidate category with the highest predicted probability in the category probability table. Step S13 removes the candidate categories related to the recognition result, that is, the pursuit of accuracy is abandoned.

[0060] S14. Taking the candidate category with the largest probability in the updated category probability table as the first approximate result of the corresponding element.

[0061] For example, a user takes a photo containing white cloud elements as the image to be processed. According to its working principle, the image recognition model will predict various alternative categories for the image elements in the image to be processed. If they are arranged from high to low according to the prediction probability, including: white clouds, marshmallows, cotton, etc., among them, the "white clouds" as the recognition result has the highest prediction probability. However, in the method provided, the alternative category "white clouds" is deliberately deleted, and the "marshmallows" with the second highest prediction probability is selected as the first approximate result to inspire the user's imagination, which is conducive to the staff engaged in literature, art and other fields to develop creative inspiration.

[0062] It can be understood that the present application discloses an image element approximation method that uses an artificial neural network-based image element recognition model to identify each element in the image to be processed, obtaining a recognition result and a category probability table for each element; wherein the recognition result is the alternative category with the highest predicted probability in the category probability table. The image element approximation method disclosed in the present application excludes the alternative categories related to the recognition result in the category probability table, and then reselects the alternative category with the highest predicted probability in the category probability table as the approximate result output. The image element approximation method disclosed in the present application abandons the traditional image recognition model's pursuit of accuracy and selects the approximate recognition results of image elements for output, which can effectively inspire users' imagination and open up their creative inspiration.

[0063] In a second aspect, the present application provides another image element approximation method, which includes:

[0064] S21. Training an image element recognition model using a locally stored facial image set, where the facial image set includes images of various facial regions of each image in a local target album.

[0065] The image element approximation method provided in this application is applicable to smart terminals such as mobile phones and tablets. The locally stored facial image set can be a facial image set from a local target album on the mobile phone or other smart terminal. The image element recognition model is trained using the facial image set, that is, the image element recognition model learns to perform facial recognition within a user-specified range. During subsequent use, the recognition results for any facial image will be those for faces that have appeared in the facial image set.

[0066] S22: Obtain an image to be processed.

[0067] The user can use the camera application to take the image to be processed, or download the image to be processed from the network. The facial elements of any person in the image to be processed are the image elements to be recognized.

[0068] S23. Recognize each face region in the image to be processed by using the image element recognition model to obtain a recognition result for each face region and a candidate person probability table, wherein the candidate person probability table includes the probability that the face region belongs to each candidate person.

[0069] The working principle and network structure of the image element recognition model mentioned in the second aspect are consistent with those of the image element recognition model mentioned in the first aspect, and will not be repeated here.

[0070] S24. Remove the candidate corresponding to the identification result in the candidate probability table, and update the candidate probability table.

[0071] The recognition result is the candidate with the highest predicted probability in the category probability table. Step S24 removes the candidate corresponding to the recognition result, that is, the pursuit of accuracy is abandoned.

[0072] S25. The candidate with the highest probability in the updated candidate probability table is used as the first approximation result of the corresponding face area.

[0073] It will be understood that the image element approximation method disclosed in the second aspect of this application trains an image element recognition model using a locally stored set of facial images, and then, according to the approximation method disclosed in the first aspect, uses the trained image element recognition model to output approximate results for each facial region in the image to be processed. After being trained on the locally stored set of facial images, the image element recognition model is capable of recognizing each locally stored portrait. When approximating the target facial region in the image to be processed, in addition to the true result, the recognition result of the person closest to the target facial region among the locally stored persons can be obtained, thereby enhancing the fun of facial recognition.

[0074] In a third aspect, the present application provides another image element approximation method, which includes:

[0075] S31. Training an image element recognition model using a target image set, where the target image set includes multiple sample images that have been classified in at least two levels.

[0076] Similar to the target image set of the target detection dataset PASCAL VOC, various image elements are classified into multiple levels. For example, the first-level classification includes animals, furniture, vehicles, etc., and the second-level classification of animals includes cats, dogs, mice, etc.

[0077] S32: Obtain an image to be processed.

[0078] The image element approximation method provided in this application is applied to smart terminals such as mobile phones and tablets. Users can use the camera application to shoot the image to be processed or download the image to be processed through the network. Any element that constitutes an object in the image to be processed is the image element to be identified.

[0079] S33. Identify each element in the image to be processed using the trained image element recognition model to obtain a recognition result and a category probability table for each element. The category probability table includes the probability that the element belongs to each candidate category. The candidate category is the final classification category of at least two levels of classification.

[0080] The working principle and network structure of the image element recognition model mentioned in the third aspect are consistent with those of the image element recognition model mentioned in the first aspect, and will not be repeated here.

[0081] S34. Remove the candidate categories in the category probability table that belong to the same upper category as the recognition result, and update the category probability table.

[0082] The recognition result is the candidate category with the highest predicted probability in the category probability table. Step S34 removes the candidate categories that belong to the same upper-level category as the recognition result, that is, the approximate results of the recognition result are abandoned to a greater extent, and the pursuit of accuracy is abandoned to a greater extent.

[0083] S35. Use the candidate category with the largest probability in the updated category probability table as the first approximate result of the corresponding element.

[0084] It is understandable that since the alternative categories belonging to the same upper-level classification as the recognition result are removed, the first approximate result has a lower correlation with the recognition result, and the pursuit of accuracy is abandoned to a greater extent. At the same time, it also inspires the user's imagination to a greater extent.

[0085] S36. Obtain an image element approximation model with the same network structure as the image element recognition model.

[0086] In the embodiment of the present application, the network structure and working principle of the image element approximation model and the image element recognition model are the same, and will not be repeated here.

[0087] S37. Remove sample images in the target image set that belong to the same upper-level category as the recognition result, and update the target image set.

[0088] S38. Train the image element approximation model using the updated target image set.

[0089] After removing sample images that belong to the same upper-level classification as the above recognition results from the target image set, the image element approximation model is trained using the remaining target image set. Then, the image element approximation model is naturally unable to recognize the true recognition results of the image elements.

[0090] S39. Recognize each element in the image to be processed by using the trained image element approximation model, and use the recognition result obtained by the image element approximation model as the second approximation result of each element.

[0091] For example, Figure 3 As shown, the user takes a photo containing white cloud elements as the image to be processed. According to the image element approximation method disclosed in the third aspect, two approximate results will be obtained: marshmallow and white rabbit, to inspire the user's imagination.

[0092] It can be understood that the third aspect of the present application discloses an image element approximation method. On the one hand, after the image element recognition model trained on the target image set recognizes each element in the image to be processed, the alternative categories in the category probability table that belong to the same upper-level classification as the recognition result are removed, and the alternative category with the largest probability is selected from the remaining category probability table as the first approximate result of the corresponding element. On the other hand, after removing the sample images that belong to the same upper-level classification as the above-mentioned recognition result from the target image set, the image element approximation model is trained with the remaining target image set. Then, the image element approximation model is naturally unable to recognize the true recognition result of the image element, and can only select the alternative category with the largest probability from other alternative categories as the second approximate result of the corresponding element. The above method obtains two approximate results by means of the image element recognition model and the image element approximation model, which can better inspire the user's imagination and open up the user's creative inspiration.

[0093] In an optional embodiment of the present application, the image element recognition model is trained using the target image set according to the following loss function:

[0094] LOSS=L 11 +L 12 +L2+L3;

[0095] in,

[0096]

[0097] Among them, λ and λ noobj is the weight corresponding to the loss factor, which is generally a preset constant; Indicates that if the j-th prediction box corresponding to the i-th grid of the feature image of the input image of the image element recognition model is responsible for predicting the image element, then the value is assigned to 1, otherwise it is assigned to 0; S 2 Represents the total number of grids in the input image, and B represents the total number of prediction boxes corresponding to each grid; (x i ,y i ) represents the real box coordinates of the image element corresponding to the i-th grid, Indicates the predicted box coordinates corresponding to the i-th grid, (w i ,h i ) represents the width and height of the ground truth box of the image element corresponding to the i-th grid, Represents the width and height of the prediction box corresponding to the i-th grid; C i represents the credibility score of the real box corresponding to the i-th grid, Represents the credibility score of the prediction box corresponding to the i-th grid; Indicates that if the j-th prediction box corresponding to the i-th grid of the feature image of the input image of the image element recognition model is not responsible for predicting the image element, then the value is assigned to 1, otherwise it is assigned to 0; Indicates that if the prediction box corresponding to the i-th grid of the feature image of the input picture of the image element recognition model contains the image element, it is assigned a value of 1, otherwise it is assigned a value of 0; i (c) represents the probability of predicting the real box element corresponding to the i-th grid of the input image as category c, It represents the probability of predicting the predicted box element corresponding to the i-th grid of the input image as category c, and classes represents the total set of category c.

[0098] In the embodiment of the present application, the network structure and working principle of the image element approximation model are the same as those of the image element recognition model, and the training method of the image element approximation model is also the same as that of the image element recognition model, which will not be repeated here.

[0099] In a fourth aspect, the present application provides a terminal device comprising one or more processors; one or more input devices; one or more output devices; and a memory. The processors, input devices, output devices, and memory are connected via a bus. The memory is configured to store a computer program comprising program instructions, and the processor is configured to execute the program instructions stored in the memory. The processor is configured to invoke the program instructions to perform the operations of any of the methods of aspects 1 to 3.

[0100] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0101] Input devices may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc. Output devices may include a display (LCD, etc.), a speaker, etc.

[0102] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0103] In specific implementations, the processor, input device, and output device described in the embodiments of the present invention can execute the implementation method described in any method of the first to third aspects, and can also execute the implementation method of the terminal device described in the embodiments of the present invention, which will not be repeated here.

[0104] In a fifth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps of any one of the methods of the first to third aspects are implemented.

[0105] The computer-readable storage medium may be an internal storage unit of the terminal device of any of the aforementioned embodiments, such as a hard disk or memory of the terminal device. The computer-readable storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device. Furthermore, the computer-readable storage medium may include both an internal storage unit and an external storage device of the terminal device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed terminal devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0108] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the embodiments of the present invention.

[0109] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0110] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0111] The terms "first," "second," "the first," or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these terms do not limit the corresponding components. The above terms are configured solely for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, even though both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0112] When one element (for example, a first element) is referred to as being “(operably or communicably) coupled” or “(operably or communicably) coupled to” or “connected to” another element (for example, a second element), it should be understood that the one element is directly connected to the other element or that the one element is indirectly connected to the other element via yet another element (for example, a third element). Conversely, it should be understood that when an element (for example, a first element) is referred to as being “directly connected” or “directly coupled” to another element (the second element), there is no element (for example, a third element) interposed therebetween.

[0113] It should be noted that, in this document, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0114] The above description is merely an optional embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

[0115] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0116] The above description is merely an optional embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

[0117] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. An image element approximation method, characterized in that: include: Training an image element recognition model using a target image set, wherein the target image set includes a plurality of sample images that have been classified at least two levels; Get the image to be processed; Identifying each element in the image to be processed using an image element recognition model to obtain a recognition result and a category probability table for each element, wherein the category probability table includes the probability that the element belongs to each candidate category; The candidate category is a final classification category of the at least two levels of classification, and the identifying each element in the image to be processed by the image element recognition model includes: identifying each element in the image to be processed by the trained image element recognition model; Removing the candidate categories related to the recognition result from the category probability table and updating the category probability table; removing the candidate categories related to the recognition result from the category probability table and updating the category probability table includes: removing the candidate categories from the category probability table that belong to the same upper-level category as the recognition result and updating the category probability table; Taking the candidate category with the largest probability in the updated category probability table as the first approximate result of the corresponding element; Acquire an image element approximation model having the same network structure as the image element recognition model; removing sample images from the target image set that belong to the same upper-level classification as the recognition result, and updating the target image set; Training the image element approximation model using the updated target image set; Each element in the image to be processed is identified by using the trained image element approximation model, and the identification result obtained by the image element approximation model is used as the second approximation result of each element.

2. The image element approximation method according to claim 1, characterized in that: The image element recognition model includes an input layer, a composite hidden layer, a prediction layer, and an output layer; The input layer is used to obtain input images; The composite hidden layer includes at least two convolution-pooling modules arranged in sequence, for extracting a feature image of the input image; The prediction layer includes two fully connected layers, which are used to predict the prediction box position information, the prediction box credibility score and the category probability table of each element in the input image according to the feature image; The output layer is used to output the prediction result of the prediction layer.

3. The image element approximation method according to claim 1, characterized in that: The training of the image element recognition model using the target image set includes: The image element recognition model is trained on the target image set according to the following loss function: <h2 style=";text-align:left;direction:ltr">LOSS=L<h2 style=";text-align:left;direction:ltr"> 11 <h2 style=";text-align:left;direction:ltr"> +L<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> +L2+L3; in, Among them, λ and λ noobj is the weight corresponding to the loss factor, which is a preset constant; Indicates that if the j-th prediction box corresponding to the i-th grid of the input image of the image element recognition model is responsible for predicting the image element, it is assigned a value of 1, otherwise it is assigned a value of 0; S 2 Represents the total number of grids in the input image, and B represents the total number of prediction boxes corresponding to each grid; (x i ,y i ) represents the real box coordinates of the image element corresponding to the i-th grid, Indicates the predicted box coordinates corresponding to the i-th grid, (w i ,h i ) represents the width and height of the ground truth box of the image element corresponding to the i-th grid, Represents the width and height of the prediction box corresponding to the i-th grid; C i represents the credibility score of the real box corresponding to the i-th grid, Represents the credibility score of the prediction box corresponding to the i-th grid; Indicates that if the j-th prediction box corresponding to the i-th grid of the input image of the image element recognition model is not responsible for predicting the image element, it is assigned a value of 1, otherwise it is assigned a value of 0; Indicates that if the prediction box corresponding to the i-th grid of the input image of the image element recognition model contains the image element, it is assigned a value of 1, otherwise it is assigned a value of 0; i (c) represents the probability of predicting the real box element corresponding to the i-th grid of the input image as category c, It represents the probability of predicting the predicted box element corresponding to the i-th grid of the input image as category c, and classes represents the total set of category c.

4. A terminal device, characterized in that: The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 3.

Images