Image recognition method and device, equipment and storage medium

Abstract

The invention discloses an image recognition method and device, equipment and a storage medium. The method comprises the steps of obtaining a target image, and determining a target mask sequence associated with an expected recognition object category of the target image from a preset mask parameter set; using the target image and the target mask sequence as input data, inputting the input data into a pre-trained target recognition model, obtaining category confidence output by the target recognition model relative to each pixel point, training the target recognition model through a set sample image pair, and obtaining category confidence output by the target recognition model relative to each pixel point; the sample image pair comprises a sample image, a sample category label and a sample mask sequence corresponding to a sample image set to which the sample image belongs; and determining a target object category included in the target image according to the confidence of each category. According to the method, the category of the target object included in the target image is determined according to the confidence of each category, so that the target recognition model can perform targeted image recognition, and the universality and flexibility of the model for image recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image recognition method, device, equipment and storage medium. Background Art

[0002] Before financial institutions provide credit products to farmers in rural areas, in order to ensure the rationality of the credit products provided, they need to verify the crop planting information provided by farmers through visits and investigations. However, this verification method has the problems of high cost, long cycle and difficulty in verification.

[0003] At present, with the development of satellite remote sensing technology, many financial institutions have adopted image recognition of remote sensing images to verify the information of crop planting in cultivated land.

[0004] However, in the process of using image recognition models for crop identification, the existing models can only extract features in the image based on the feature extraction module and cannot perform targeted identification. In addition, the model is trained only by using different data sets to perform corresponding training on the model to complete different feature extraction tasks, resulting in low versatility, flexibility and accuracy of the image recognition model. Summary of the invention

[0005] The present invention provides an image recognition method, device, equipment and storage medium to solve the problems that the image recognition model cannot perform targeted recognition and has low versatility, flexibility and accuracy.

[0006] In a first aspect, an embodiment of the present invention provides an image recognition method, the method comprising:

[0007] Acquire a target image, and determine a target mask sequence associated with an object category desired to be identified by the target image from a preset mask parameter set;

[0008] The target image and the target mask sequence are used as input data to a pre-trained target recognition model, and the category confidence of the target recognition model relative to each pixel point output is obtained, wherein the target recognition model is trained by a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to a sample image set to which the sample image belongs;

[0009] According to each of the category confidences, a target object category included in the target image is determined.

[0010] In a second aspect, an embodiment of the present invention provides an image recognition device, the device comprising:

[0011] A mask sequence acquisition module, used to acquire a target image and determine a target mask sequence associated with an object category desired to be identified in the target image from a preset mask parameter set;

[0012] A confidence determination module, used to input the target image and the target mask sequence as input data into a pre-trained target recognition model, and obtain the category confidence of the target recognition model relative to each pixel output, wherein the target recognition model is trained by a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to a sample image set to which the sample image belongs;

[0013] The object category determination module is used to determine the target object category included in the target image according to each of the category confidences.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0015] at least one processor;

[0016] and a memory communicatively coupled to the at least one processor;

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image recognition method described in any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image recognition method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention obtains a target image and determines a target mask sequence associated with the category of the object expected to be identified by the target image from a preset mask parameter set; the target image and the target mask sequence are input as input data to a pre-trained target recognition model, and the category confidence of the target recognition model relative to the output of each pixel is obtained. The target recognition model is trained by a set sample image pair, and the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to the sample image set to which the sample image belongs; according to each category confidence, the category of the target object included in the target image is determined. This method inputs the target image and the target mask sequence into a pre-trained target recognition model to obtain the category confidence of the target recognition model relative to the output of each pixel, and determines the category of the target object included in the target image according to each category confidence, so that the target recognition model can perform targeted image recognition to meet different recognition requirements for the same target image, thereby improving the versatility and flexibility of the target recognition model.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an image recognition method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] It should be noted that in the process of using the image recognition model for crop recognition, the existing model can only extract features from the image based on the feature extraction module, and cannot perform targeted recognition. For example, if the trained image recognition model can recognize crops such as soybeans, corn and rice, it is impossible to recognize only one or two of the above three crops when using the image recognition model. And the way to train the model is only to use different data sets to perform corresponding training on the model that completes different feature extraction tasks, resulting in low versatility, flexibility and accuracy of the image recognition model. For example, if model one wants to recognize soybeans and corn, it is necessary to train model one with a data set containing soybeans and corn. If model two wants to recognize rice, it is necessary to train model two with a data set containing rice. If model three wants to recognize soybeans, corn and rice, it is necessary to train model three with a data set containing soybeans, corn and rice. It is impossible to train model three with data set one and data set two because there is a conflict between the background labels of data set one and data set two, that is, the background label of data set one contains rice, and the background label of data set two contains soybeans and corn.

[0028] Based on this, an embodiment of the present invention provides an image recognition method. Figure 1 A flowchart of an image recognition method provided in an embodiment of the present invention. The embodiment of the present invention is applicable to scenarios in which image recognition is performed using an image recognition model. The method can be executed by an image recognition device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which is preferably a mobile terminal, a desktop computer, a laptop computer, a server, etc.

[0029] like Figure 1 As shown, the image recognition method provided by the embodiment of the present invention may specifically include:

[0030] S101, acquiring a target image, and determining a target mask sequence associated with an object category desired to be identified by the target image from a preset mask parameter set.

[0031] Among them, the target image can be understood as one or more images that meet the image recognition requirements selected after the image recognition operation is started. Exemplarily, the target image can be an image directly captured by an image capture device on the execution device, or it can be an image selected from images stored by the execution device. The original image is usually saved with high resolution and high color depth, and can be saved in a lossless image format, such as RAW format, TIFF format, or PNG format, so as to retain as much image information as possible in the original image. The target mask sequence can be understood as the mask sequence corresponding to the object category that is expected to be identified. The mask parameter set can be understood as a set consisting of each object category and each mask parameter corresponding to each object category.

[0032] Exemplarily, when there are three object categories, namely soybeans, corn and rice, the mask parameter set may be a set of possible mask parameters corresponding to all object categories. When it is determined that the target image expects the object category to be identified as soybeans, the associated target mask sequence is [1, 0, 0]; when the expected object category to be identified is corn, the associated target mask sequence is [0, 1, 0]; when the expected object category to be identified is rice, the associated target mask sequence is [0, 0, 1]; when the expected object category to be identified is soybeans and corn, the associated target mask sequence is [1, 1, 0]; when the expected object category to be identified is soybeans and rice, the associated target mask sequence is [1, 0, 1]; when the expected object category to be identified is corn and rice, the associated target mask sequence is [0, 1, 1]; when the expected object category to be identified is soybeans, corn and rice, the associated target mask sequence is [1, 1, 1].

[0033] In this embodiment, the target image can be acquired by directly capturing it through a capture device, or by selecting or intercepting it from a stored image. The embodiment of the present invention does not limit the method of acquiring the target image, and according to the object category desired to be identified in the target image, the associated mask parameters are searched from a preset mask parameter set and the mask parameters are determined as a target mask sequence.

[0034] S102. The target image and the target mask sequence are input as input data to a pre-trained target recognition model, and the category confidence of the target recognition model relative to each pixel output is obtained. The target recognition model is trained by a set sample image pair, and the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to the sample image set to which the sample image belongs.

[0035] Among them, the target recognition model can be understood as a model that can identify the category to which the object to be recognized in the target image belongs. The target recognition model can adopt a neural network model such as a convolutional neural network model, a recurrent neural network model, a generative adversarial network model or a long short-term memory network model. The category confidence can be understood as the set of probabilities that the target recognition model believes that each pixel in the target image belongs to each object category. For example, if the object categories corresponding to a certain target image are soybean, corn and rice, then the category confidence corresponding to a certain pixel in the target image can be [78%, 48%, 2%], that is, the target recognition model believes that the probability that the pixel belongs to soybean is 78%, the probability that it belongs to corn is 48%, and the probability that it belongs to rice is 2%.

[0036] In this embodiment, the sample image can be understood as an image that has been manually or automatically annotated, which is used to train the target recognition model for object recognition and positioning. It should be noted that the sample image can come from one or more sample image sets. The sample category label can be understood as a set of category labels corresponding to each pixel in each sample image. Each category label in the sample category label can correspond to each pixel in each sample image by one-to-one correspondence with the arrangement position of the pixel in the sample image, or by establishing an association table. The sample mask sequence can be understood as a unique mask sequence corresponding to the sample image set composed of the object categories contained in each sample image set, which is used to help the target recognition model perform targeted recognition.

[0037] In this embodiment, the target image and the target mask sequence are input into a trained target recognition model. The target recognition model determines the confidence of each pixel corresponding to each object category based on the features of each pixel in the extracted target image. Then, by combining the target mask sequence, the category confidence output by the target recognition model corresponding to the desired object category is obtained.

[0038] Exemplarily, the trained target recognition model can recognize soybeans, corn and rice. When the target mask sequence is selected as [1, 1, 0], it is considered that the object categories that the target recognition model is expected to recognize are soybeans and corn. When the target image and the target mask sequence are input into the trained target recognition model, the target recognition model obtains the confidence of each object category corresponding to each pixel in the target image according to the input target image, such as obtaining the confidence of each object category corresponding to a certain pixel as [88%, 10%, 3%], and then combines the input target mask sequence to select the confidence value at the position corresponding to the target mask sequence with a value of 1 from the confidence of each object category, that is, [88%, 10%], that is, the confidence value of soybeans and corn, and uses it as the category confidence output by the model.

[0039] In this embodiment, the target recognition model is trained by using a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to a sample image set to which the sample image belongs.

[0040] S103: Determine the target object category included in the target image according to the confidence level of each category.

[0041] The target object may be understood as one or more objects contained in the target image that are expected to be identified.

[0042] In this embodiment, based on the category confidence corresponding to each pixel point, the category corresponding to each category confidence is determined by a given rule, that is, the category corresponding to each pixel point is determined, and then the target object category included in the target image is determined based on the distribution of the pixel points.

[0043] The method of determining the category corresponding to each category confidence by a given rule can be to first determine whether there is a confidence value exceeding a given threshold among all the confidence values ​​contained in the category confidence; if so, determine that the category corresponding to the maximum confidence value exceeding the given threshold is the category of the corresponding pixel point; if not, determine that the pixel point is the background.

[0044] For example, continuing with the above example, if the category confidence output by the model is [88%, 10%], the categories corresponding to each confidence value are soybean and corn, respectively. When the given threshold is 50%, the category of the corresponding pixel point is determined to be soybean.

[0045] For example, if the category confidence output by the model is [8%, 10%], the categories corresponding to the confidence values ​​are soybean and corn, respectively, and when the given threshold is 20%, the category of the corresponding pixel is determined to be background.

[0046] An image recognition method provided by an embodiment of the present invention obtains a target image and determines a target mask sequence associated with the category of the object expected to be recognized by the target image from a preset mask parameter set; the target image and the target mask sequence are input as input data to a pre-trained target recognition model, and the category confidence of the target recognition model output relative to each pixel is obtained, and the target recognition model is trained by a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to the sample image set to which the sample image belongs; according to each category confidence, the category of the target object included in the target image is determined. The method inputs the target image and the target mask sequence into a pre-trained target recognition model to obtain the category confidence of the target recognition model output relative to each pixel, and according to each category confidence, the category of the target object included in the target image is determined, so that the target recognition model can perform targeted image recognition to meet different recognition requirements for the same target image, thereby improving the versatility and flexibility of the target recognition model.

[0047] It is understandable that after determining the target object categories included in the target image according to the confidence of each category, different target object categories can be displayed in different colors to help promote subsequent applications. For example, when verifying whether the crop information filled in by farmers when applying for credit products is true or not, the target image can be captured by remote sensing equipment, image recognition can be performed on the target image, and the expected target object category can be obtained according to the target mask sequence. At the same time, different colors are used to represent each target object category, which can intuitively display the types of crops and area contained in the farmer's cultivated land, thereby improving the efficiency and accuracy of verification and reducing the cost and difficulty of verification.

[0048] As a first optional embodiment of this embodiment, based on the above embodiment, the step of determining the target mask sequence associated with the object category expected to be recognized in the target image from the preset mask parameter set can be specifically implemented as the following steps:

[0049] a1) determining the category of the object to be identified from the target image.

[0050] The category of the object to be identified can be understood as the category of the object to be identified from the target image. It can be different categories of a certain type of object, such as different types of crops, different types of animals, different types of fruits, etc. It can also be different types of objects, such as people, vehicles, animals, etc. It should be noted that the type of the object to be identified cannot be semantically ambiguous, such as mouse refers to a mouse in one dataset and a mouse in another dataset.

[0051] In this embodiment, the category of the object to be identified that is expected to be identified from the target image is determined according to the requirements. The category of the object to be identified may include one or multiple categories at the same time. For example, the category of the object to be identified may be soybeans or soybeans and corn. One or more categories of the object to be identified may be determined at one time. For example, the first object to be identified is determined to be soybeans and corn, and the second object to be identified is determined to be rice.

[0052] b1) searching for a preset mask parameter set to determine a target mask sequence, wherein the target mask sequence includes mask parameters corresponding to all categories of objects to be identified.

[0053] In this embodiment, mask parameters corresponding to all types of objects to be identified are searched in a preset mask parameter set, and the mask parameters are determined as a target mask sequence, that is, the target mask sequence includes mask parameters corresponding to all types of objects to be identified.

[0054] The above technical solution of this embodiment provides strong support for subsequent targeted identification by searching for a preset mask parameter set according to the category of the object to be identified and determining a target mask sequence.

[0055] As a second optional embodiment of this embodiment, the step of determining the target object category included in the target image according to each category confidence level may be specifically optimized into the following steps:

[0056] a2) dividing at least one confidence information contained in each category confidence according to a given division rule to determine each target confidence.

[0057] Among them, the confidence information can be understood as the confidence value relative to each category in the category confidence. Exemplarily, the category confidence is [88%, 10%], and 88% and 10% are the two confidence information contained in the category confidence. The target confidence can be understood as the confidence information corresponding to a certain category, which is used to determine the final category of the corresponding pixel point.

[0058] In this embodiment, at least one confidence information contained in each category confidence output by the target recognition model relative to each pixel point is divided according to a given division rule, and the division result is determined as the target confidence corresponding to the category confidence.

[0059] Exemplarily, a given division rule may be to first determine whether any confidence information contained in each category confidence exceeds a set threshold; if there is no confidence information exceeding the set threshold in the category confidence, then the target confidence corresponding to the category confidence is determined to be 0%; if there is confidence information exceeding the set threshold, then the confidence information with the largest value among the confidence information is determined as the target confidence; for example, if the set threshold is 50% and the category confidence is [20%, 70%, 95%], then 95% is determined as the target confidence.

[0060] b2) Aggregating pixel points corresponding to target confidences of the same category, and forming a category image region based on the pixel points.

[0061] The category image region can be understood as an image region belonging to a specific category formed by the aggregation of pixels of the same category. For example, if all the pixels in an area identified by the target recognition model belong to the category of "soybeans", then these pixels will be aggregated to form a category image region representing the category of "soybeans"; if all the pixels in an area identified by the target recognition model belong to the category of "cats", then these pixels will be aggregated to form a category image region representing the category of "cats".

[0062] In this embodiment, the pixel points corresponding to the target confidence of the same category are aggregated together to form the category image area belonging to each category. The target confidence of the same category can be understood as classifying the target confidence according to the sequence number or position of the target confidence in the category confidence. For example, if each category confidence in the target image contains three confidence information, the target confidence can be divided into three categories according to the three situations that the target confidence is in the first, second and third place in the category confidence.

[0063] c2) determining the image category corresponding to the target confidence as the target object category of the corresponding category image area.

[0064] In this embodiment, the image category corresponding to each pixel is determined according to the target confidence, so as to determine the target object category of the category image area composed of the pixel points.

[0065] It should be noted that the correspondence rules between the target confidence and the image category corresponding to each pixel need to be set in advance. For example, if the target confidence is 0%, the image category corresponding to the pixel is background, and if the target confidence is not 0%, the object category corresponding to the target confidence is determined as the image category, and the image category is determined as the target object category of the category image area composed of pixels corresponding to the same type of target confidence (i.e., the same type of image category).

[0066] The above technical scheme of this embodiment determines the confidence of each target by dividing the confidence information in each category confidence, and aggregates the pixel points corresponding to the same category of target confidence to form a category image area, and determines the image category corresponding to the target confidence as the target object category of the corresponding category image area, thereby realizing effective recognition of the target object category in the image and positioning of the target object.

[0067] As a third optional embodiment of this embodiment, based on the above embodiment, the step of determining the mask parameter set can be concretized into the following steps:

[0068] a3) obtaining at least one sample image set, determining the object categories of the objects contained in the sample images of each of the sample image sets, and counting the total number of the object categories.

[0069] In this embodiment, at least one sample image set is obtained after image annotation to determine the category to which each pixel point of each sample image in the sample image set belongs, the object categories of different objects contained in the sample images of each of the sample image sets are determined, and the total number of object categories is counted.

[0070] It should be noted that the object categories of the included objects are all categories obtained based on the image annotation information, including background object categories and non-background object categories, and the total number of statistical object categories only includes the number of non-repeated non-background object categories.

[0071] b3) determining a pixel point label sequence corresponding to each pixel point of the sample image according to each object category and a set encoding method, wherein a first sequence length of the pixel point label sequence is the total amount of the category.

[0072] The pixel label sequence can be understood as a label in the form of an array sequence used to indicate which category the pixel belongs to. The first sequence length can be understood as the length of the pixel label sequence.

[0073] In this embodiment, according to each object category and a preset encoding method, a corresponding pixel label sequence is assigned to each pixel in the sample image, and the pixel label sequence is a one-dimensional array, and the array length (i.e., the first sequence length) is equal to the total number of categories.

[0074] As one implementation method, this optional embodiment may further specify the pixel label sequence corresponding to the pixel of each sample image according to each object category and the set encoding method as follows:

[0075] b31) Sort the object categories, obtain the sorting sequence number of each object category, and create an initial label sequence with a sequence length equal to the total number of categories and an initial encoding value of 0.

[0076] In this embodiment, the object categories of the objects contained in the sample images of each sample image set are summarized, and the summarized object categories are deduplicated, and then the deduplicated object categories are sorted to obtain the sorting sequence number of each object category, wherein the sorting sequence number starts from 1 and is sorted by accumulating 1 each time, that is, the sorting sequence number can be 1, 2, 3, ... and so on. At the same time, an initial label sequence with a sequence length equal to the total number of categories is created, and the initial value of each encoding value in the initial label sequence is 0.

[0077] b32) For each sample image in each sample image set, select a sample object category contained in the sample image from the object categories.

[0078] The sample image category may be understood as the object category selected and included in each sample image in the object category.

[0079] In this embodiment, a sample object category is selected from the object categories for each sample image in each sample image set.

[0080] b33) For each pixel in the sample image, determine the pixel object category corresponding to the pixel according to the position information of the sample object category.

[0081] The position information may be understood as the corresponding area in the sample image.

[0082] In this embodiment, for each pixel in the sample image, the pixel object category corresponding to each pixel is determined according to the position information of the sample object category.

[0083] Exemplarily, the sample object categories are soybeans and corn, and the location information of the sample object categories is the distribution area of ​​soybeans in the target image and the distribution area of ​​corn in the target image. It can be understood that other areas where neither soybeans nor corn are distributed are background areas. The pixel object category corresponding to the pixel points in each area can be determined based on the location information of the sample object category.

[0084] b34) Determine the first sequence number associated with the pixel point object category in the initial label sequence according to the sample sorting sequence number of each sample object category.

[0085] The sample sorting number can be understood as the sorting number obtained when the sample object category is sorted. The first sequence number can be understood as the encoding position of the corresponding pixel object category in the initial label sequence. It should be noted that the first sequence number only corresponds to one encoding position.

[0086] In this embodiment, according to the sample sorting sequence numbers of the sample object categories, the first sequence number associated with the pixel object category is determined in the initial label sequence according to the sample sorting sequence number.

[0087] Exemplarily, if the pixel object category of a certain pixel in the sample image is corn and the sample ranking of corn is 2, then the first sequence number in the initial label sequence associated with the pixel object category (corn) is the second bit code in the initial label sequence.

[0088] It should be noted that if the sample object category corresponding to the pixel point is background, that is, there is no sample sorting number, the encoding values ​​of all positions in the initial label sequence can be directly set to 0 and the initial label sequence can be determined as the pixel point label sequence of the pixel point, and steps b35) and b36) are no longer entered.

[0089] b35) setting the code value corresponding to the first sequence number in the initial tag sequence to 1, and setting the code values ​​at other positions to 0.

[0090] In this embodiment, the code value corresponding to the first sequence number in the initial tag sequence is set to 1, and the code values ​​at other positions are set to 0.

[0091] Exemplarily, if the total number of categories is 5, the sequence length of the initial label sequence is 5, and the initial label sequence is [0,0,0,0,0]. Continuing with the above example, if the pixel object category of the pixel in the sample image is corn and the sample ranking of corn is 2, then the encoding value corresponding to the first serial number in the initial label sequence corresponding to the pixel is set to 1, that is, the initial label sequence is [0,1,0,0,0].

[0092] b36) Determine the updated initial label sequence as the pixel label sequence of the pixel point.

[0093] In this embodiment, the updated initial label sequence is determined as the pixel label sequence of the pixel points, that is, the pixel label sequence is used to represent the object category of the pixel points.

[0094] The above technical scheme of this embodiment, by creating an initial label sequence and obtaining the sorting number of each object category, realizes the efficient, convenient and low-cost allocation of pixel label sequences to each pixel in each sample image, so that each pixel can obtain a new label form without relying on manual annotation, so as to solve the problem that the background labels between the sample image sets cannot be merged into one label due to the different semantics, and thus cannot participate in the training of the same target recognition model at the same time.

[0095] c3) determining a mask sequence of the sample image set according to a pixel label sequence of each sample image in each sample image set, wherein a second sequence length of the mask sequence is the same as the first sequence length, and the mask sequence includes encoding values ​​of each object category associated with the sample image set.

[0096] The second sequence length can be understood as the length of the mask sequence, which is the same as the first sequence length.

[0097] In this embodiment, a mask sequence corresponding to the sample image set is determined based on a pixel label sequence of each sample image in each sample image set, wherein the second sequence length of the mask sequence is the same as the first sequence length (i.e., the total number of categories), and the mask sequence includes the encoding values ​​of each object category associated with the sample image set, i.e., the object category information associated with the sample image set can be obtained through the mask sequence.

[0098] As one implementation manner, this optional embodiment may further specify the determination of the mask sequence of the sample image set according to the pixel label sequence of each sample image in each sample image set as follows:

[0099] c31) For each sample image set, determining a pixel point label sequence contained in each sample image in the sample image set.

[0100] In this embodiment, for each sample image set, the pixel label sequence contained in each sample image in the sample image set is determined. Exemplarily, the pixel label sequence contained in each sample image in a sample image set may be [1, 0, 0], [0, 1, 0], [0, 0, 0].

[0101] c32) performing a union operation on the pixel point label sequences, and determining the operation result as the mask sequence of the sample image set.

[0102] In this embodiment, a union operation is performed on the pixel point label sequence, that is, corresponding operations are performed on the coding values ​​at corresponding positions in the pixel point label sequence, and the operation rule is that if there is at least one 1 in the coding value at the corresponding position, the coding value of the position in the obtained operation result is 1, and the operation result obtained after the coding values ​​at each position in the pixel point label sequence are operated is determined as the mask sequence of the sample image set.

[0103] Continuing with the above example description, when the pixel label sequence is [1,0,0], [0,1,0], [0,0,0], the result after the union operation is [1,1,0], that is, the mask sequence of the sample image set is [1,1,0].

[0104] The above technical scheme of this embodiment obtains the mask sequence of each sample image set by performing a union operation on the pixel point label sequence contained in each sample image set, so that the target recognition model can be trained using each sample image set, and the prediction of the corresponding sample image set is realized according to the mask sequence of each sample image set, thereby maximizing the use of different sample image sets to meet more image recognition needs.

[0105] d3) constructing the mask parameter set based on the determined mask sequences.

[0106] In this embodiment, the object category of each sample image set and each mask sequence corresponding to each determined sample image set are correspondingly stored to form a mask parameter set.

[0107] In addition, the mask parameter set may also store the correspondence between the object category of each pixel and the corresponding pixel label sequence.

[0108] The above technical scheme of the present embodiment determines the first sequence length of the pixel label sequence and the second sequence length of the mask sequence according to the object category in the acquired image sample set, and at the same time determines the pixel label sequence corresponding to the pixel points of each sample image according to each object category and the set encoding method, determines the mask sequence of the sample image set according to the pixel label sequence, and implements the construction of the mask parameter set based on each mask sequence, so that the target recognition model can output the corresponding prediction result of the corresponding data set, thereby improving the flexibility and versatility of the target recognition model.

[0109] As one implementation of any embodiment of the present invention, the training steps of the target recognition model can be further specified as follows:

[0110] a4) obtaining a sample image from at least one sample image set, and obtaining a sample category label and a sample mask sequence pre-determined for each of the sample images to form a sample image pair, wherein the sample category label includes a pixel point label sequence corresponding to each pixel point.

[0111] In this embodiment, at least one sample image is obtained from a sample image set stored in an execution device or on a cloud service platform by reading, sampling, etc., and a sample category label pre-determined for each of the sample images and a sample mask sequence (i.e., a result of a union operation on the pixel point label sequence) are obtained. The sample category label includes a pixel point label sequence corresponding to each pixel point. Then, each sample image, each sample category label and each sample mask sequence are matched to form a sample image pair.

[0112] b4) Inputting the sample image in the sample image pair and the sample mask sequence as sample data into the created initial recognition model to obtain the actual pixel confidence output by the initial recognition model for the pixel in the sample image.

[0113] The sample data can be understood as data used to input the initial recognition model for model training. The initial recognition model can be understood as an untrained target recognition model. The actual pixel confidence can be understood as the prediction confidence output by the initial recognition model for the pixel in each sample image, which is used to help the initial recognition model determine the object category of the pixel.

[0114] In this embodiment, the sample images in the sample image pair and the sample mask sequence are used as sample data and are input into the created initial recognition model pair by pair to obtain the actual pixel confidence for each pixel in each sample image output by the initial recognition model according to the sample mask sequence.

[0115] Exemplarily, the sample mask sequence corresponding to the sample image includes a sample mask sequence A[1,1,0] and a sample mask sequence B[0,0,1]. After the sample image and the sample mask sequence are input into the initial recognition model, the sample image can obtain all object categories corresponding to the sample category labels through the initial recognition model. When the object categories corresponding to the sample category labels are soybeans, corn and rice, the initial recognition model can obtain the confidence relative to each object category, that is, [soybean confidence, corn confidence, rice confidence]; then the initial recognition model obtains the actual pixel confidence of a certain pixel according to the sample mask sequence in the input data, which are the actual pixel confidence A[soybean confidence, corn confidence] and the actual pixel confidence B[rice confidence], that is, the prediction result corresponding to the sample image set to which the sample image belongs is obtained.

[0116] Continuing with the above description, according to the different actual pixel confidences obtained by inputting different sample mask sequences, rice can be accurately identified as the background in the sample image set in which the initial recognition model is expected to be trained to identify soybeans and corn. Similarly, in the sample image set in which the initial recognition model is expected to be trained to identify rice, soybeans and corn can be accurately identified as the background, and the output actual pixel confidence can correspond to the sample category label contained in each sample image, thereby effectively training the initial recognition model, so that the trained initial recognition model (i.e., the target recognition model) can accurately identify any object category among soybeans, corn and rice.

[0117] c4) determining a loss function value based on the sample category label and the confidence of each actual pixel point in combination with a set loss function.

[0118] In this embodiment, the loss function value is determined according to the label content corresponding to the actual pixel confidence in the sample category label and the actual pixel confidence, combined with a set loss function, such as a cross entropy function.

[0119] d4) updating the network parameters of the initial recognition model according to the loss function value, and returning to re-input the sample data until the iteration end condition is reached.

[0120] In this embodiment, the loss function value is returned to the initial recognition model, so that the initial recognition model updates the network parameters according to the returned loss function value, and returns to re-input sample data and executes steps b4)-d4) to train the initial recognition model when the iteration end condition is not met.

[0121] e4) determining the initial recognition model obtained after the iteration as the target recognition model.

[0122] In this embodiment, when the iteration is completed, the obtained initial recognition model is determined as the target recognition model, that is, the training of the initial recognition model is considered to be completed.

[0123] The above technical scheme of this embodiment, by using sample image sample category labels and sample mask sequences to form sample image pairs, and inputting the sample images in the sample image pairs and the sample mask sequences as sample data into the initial recognition model, obtains the actual pixel confidence output by the initial recognition model for the pixel in the sample image, and determines the loss function according to the sample category labels and the actual pixel confidences to update the network parameters of the initial recognition model, thereby realizing the simultaneous training of the initial recognition network model using data sets with different labels, thereby improving the accuracy, flexibility and versatility of the initial recognition network model.

[0124] Figure 2 FIG. 1 is a schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention. Figure 2 As shown, the device includes: a mask sequence acquisition module 21, a confidence determination module 22 and an object category determination module 23, wherein:

[0125] The mask sequence acquisition module 21 is used to acquire a target image and determine a target mask sequence associated with an object category desired to be identified in the target image from a preset mask parameter set;

[0126] A confidence determination module 22 is used to input the target image and the target mask sequence as input data to a pre-trained target recognition model, and obtain the category confidence of the target recognition model relative to each pixel point output, wherein the target recognition model is trained by a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to a sample image set to which the sample image belongs;

[0127] The object category determination module 23 is used to determine the target object category included in the target image according to each category confidence.

[0128] An image recognition device provided by an embodiment of the present invention obtains a target image and determines a target mask sequence associated with the category of the object expected to be recognized by the target image from a preset mask parameter set; the target image and the target mask sequence are input as input data to a pre-trained target recognition model, and the category confidence of the target recognition model output relative to each pixel is obtained, and the target recognition model is trained by a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to the sample image set to which the sample image belongs; according to each category confidence, the category of the target object included in the target image is determined. The method inputs the target image and the target mask sequence into the pre-trained target recognition model to obtain the category confidence of the target recognition model output relative to each pixel, and according to each category confidence, determines the category of the target object included in the target image, so that the target recognition model can perform targeted image recognition to meet different recognition requirements for the same target image, thereby improving the versatility and flexibility of the target recognition model.

[0129] Furthermore, the mask sequence acquisition module 21 may be specifically used for:

[0130] Determining a category of an object to be identified that is desired to be identified from the target image;

[0131] A preset mask parameter set is searched to determine a target mask sequence, wherein the target mask sequence includes mask parameters corresponding to all categories of objects to be identified.

[0132] Furthermore, the object category determination module 23 may be specifically used for:

[0133] Dividing at least one confidence information contained in each of the category confidences according to a given division rule to determine each target confidence;

[0134] Aggregate pixel points corresponding to the target confidence of the same category, and form a category image area based on the pixel points;

[0135] The image category corresponding to the target confidence is determined as the target object category of the corresponding category image area.

[0136] Furthermore, the image recognition device further includes a mask parameter set determination module, which may specifically include:

[0137] a category determination unit, configured to obtain at least one sample image set, determine the object category of the objects contained in the sample images of each of the sample image sets, and count the total number of the object categories;

[0138] A pixel label sequence determining unit, used to determine a pixel label sequence corresponding to the pixel points of each sample image according to each object category and a set encoding method, wherein the first sequence length of the pixel label sequence is the total amount of the category;

[0139] a mask sequence determining unit, configured to determine a mask sequence of the sample image set according to a pixel point label sequence of each sample image in each sample image set, wherein a second sequence length of the mask sequence is the same as the first sequence length, and the mask sequence includes a coding value of each object category associated with the sample image set;

[0140] A parameter set determination unit is used to form the mask parameter set based on the determined mask sequences.

[0141] Furthermore, the pixel label sequence determination unit may be specifically used for:

[0142] Sort the object categories, obtain the sorting sequence number of each object category, and create an initial label sequence with a sequence length equal to the total number of categories and an initial encoding value of 0;

[0143] For each sample image in each sample image set, selecting a sample object category included in the sample image from the object categories;

[0144] For each pixel in the sample image, determining a pixel object category corresponding to the pixel according to the position information of the sample object category;

[0145] Determine, according to the sample sorting sequence number of each of the sample object categories, a first sequence number associated with the pixel point object category in the initial label sequence, and a second sequence number associated with other sample object categories except the pixel point object category;

[0146] Set the code value corresponding to the first sequence number in the initial tag sequence to 1, and set the code values ​​at other positions to 0;

[0147] The updated initial label sequence is determined as the pixel label sequence of the pixel points.

[0148] Furthermore, the mask sequence determination unit may be specifically used for:

[0149] For each sample image set, determining a pixel point label sequence contained in each sample image in the sample image set;

[0150] A union operation is performed on the pixel point label sequences, and the operation result is determined as the mask sequence of the sample image set.

[0151] Furthermore, the image recognition device also includes a model training module, which can be specifically used for:

[0152] Acquire a sample image from at least one sample image set, and obtain a sample category label and a sample mask sequence pre-determined for each of the sample images to form a sample image pair, wherein the sample category label includes a pixel point label sequence corresponding to each pixel point;

[0153] Inputting the sample image in the sample image pair and the sample mask sequence as sample data into the created initial recognition model to obtain the actual pixel confidence output by the initial recognition model for the pixel in the sample image;

[0154] Determine the loss function value according to the sample category label and the confidence of each actual pixel point in combination with the set loss function;

[0155] The network parameters of the initial recognition model are updated according to the loss function value, and the sample data input operation is repeated until the iteration end condition is reached;

[0156] The initial recognition model obtained after the iteration is completed is determined as the target recognition model.

[0157] The image recognition device provided in the embodiment of the present invention can execute the image recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0158] Figure 3 A schematic diagram of the structure of an electronic device 30 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0159] like Figure 3 As shown, the electronic device 30 includes at least one processor 31, and a memory connected to the at least one processor 31, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 31 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 32 or the computer program loaded from the storage unit 38 to the random access memory (RAM) 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other through a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0160] A number of components in the electronic device 30 are connected to the I / O interface 35, including: an input unit 36, such as a keyboard, a mouse, etc.; an output unit 37, such as various types of displays, speakers, etc.; a storage unit 38, such as a disk, an optical disk, etc.; and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0161] The processor 31 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 31 performs the various methods and processes described above, such as an image recognition method.

[0162] In some embodiments, the image recognition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 38. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 30 via the ROM 32 and / or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the image recognition method described above may be performed. Alternatively, in other embodiments, the processor 31 may be configured to perform the image recognition method in any other appropriate manner (e.g., by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The computer programs for implementing the image recognition method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0168] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0170] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image recognition method, characterized in that: include: Acquire a target image, and determine a target mask sequence associated with an object category desired to be identified by the target image from a preset mask parameter set; The target image and the target mask sequence are used as input data to a pre-trained target recognition model, and the category confidence of the target recognition model relative to each pixel point output is obtained, wherein the target recognition model is trained by a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to a sample image set to which the sample image belongs; According to each of the category confidences, a target object category included in the target image is determined.

2. The method according to claim 1, characterized in that: The step of determining a target mask sequence associated with an object category expected to be recognized by the target image from a preset mask parameter set includes: Determining a category of an object to be identified that is desired to be identified from the target image; A preset mask parameter set is searched to determine a target mask sequence, wherein the target mask sequence includes mask parameters corresponding to all categories of objects to be identified.

3. The method according to claim 1, characterized in that Determining the target object category included in the target image according to each category confidence includes: Dividing at least one confidence information contained in each of the category confidences according to a given division rule to determine each target confidence; Aggregate pixel points corresponding to the target confidence of the same category, and form a category image area based on the pixel points; The image category corresponding to the target confidence is determined as the target object category of the corresponding category image area.

4. The method according to claim 1, characterized in that The step of determining the mask parameter set includes: Acquire at least one sample image set, determine the object category of the objects contained in the sample images of each of the sample image sets, and count the total number of the object categories; Determine a pixel point label sequence corresponding to a pixel point of each sample image according to each object category and a set encoding method, wherein a first sequence length of the pixel point label sequence is the total amount of the category; Determining a mask sequence of the sample image set according to a pixel point label sequence of each sample image in each sample image set, wherein a second sequence length of the mask sequence is the same as the first sequence length, and the mask sequence includes encoding values ​​of each object category associated with the sample image set; The mask parameter set is formed based on the determined mask sequences.

5. The method according to claim 4, characterized in that The step of determining a pixel label sequence corresponding to a pixel of each sample image according to each object category and a set encoding method includes: Sort the object categories, obtain the sorting sequence number of each object category, and create an initial label sequence with a sequence length equal to the total number of categories and an initial encoding value of 0; For each sample image in each sample image set, selecting a sample object category included in the sample image from the object categories; For each pixel in the sample image, determining a pixel object category corresponding to the pixel according to the position information of the sample object category; Determining, according to the sample sorting sequence number of each of the sample object categories, a first sequence number associated with the pixel point object category in the initial label sequence; Set the code value corresponding to the first sequence number in the initial tag sequence to 1, and set the code values ​​at other positions to 0; The updated initial label sequence is determined as the pixel label sequence of the pixel points.

6. The method according to claim 4, characterized in that Determining the mask sequence of the sample image set according to the pixel label sequence of each sample image in each sample image set includes: For each sample image set, determining a pixel point label sequence contained in each sample image in the sample image set; A union operation is performed on the pixel point label sequences, and the operation result is determined as the mask sequence of the sample image set.

7. The method according to any one of claims 1 to 6, characterized in that: The training steps of the target recognition model include: Acquire a sample image from at least one sample image set, and obtain a sample category label and a sample mask sequence pre-determined for each of the sample images to form a sample image pair, wherein the sample category label includes a pixel point label sequence corresponding to each pixel point; Inputting the sample image in the sample image pair and the sample mask sequence as sample data into the created initial recognition model to obtain the actual pixel confidence output by the initial recognition model for the pixel in the sample image; Determine the loss function value according to the sample category label and the confidence of each actual pixel point in combination with the set loss function; The network parameters of the initial recognition model are updated according to the loss function value, and the sample data input operation is repeated until the iteration end condition is reached; The initial recognition model obtained after the iteration is completed is determined as the target recognition model.

8. An image recognition device, characterized in that: include: A mask sequence acquisition module, used to acquire a target image and determine a target mask sequence associated with an object category desired to be identified in the target image from a preset mask parameter set; A confidence determination module, used to input the target image and the target mask sequence as input data into a pre-trained target recognition model, and obtain the category confidence of the target recognition model relative to each pixel output, wherein the target recognition model is trained by a set sample image pair, wherein the sample image pair includes a sample image, a sample category label, and a sample mask sequence corresponding to a sample image set to which the sample image belongs; The object category determination module is used to determine the target object category included in the target image according to each of the category confidences.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image recognition method according to any one of claims 1 to 7 when executed.

Images