Image processing method, device and computer equipment
By using the mutual learning method of feature extraction network and auxiliary training network in image processing, combined with distance parameters and threshold screening, the accuracy problem of fine-grained object category recognition in image recognition is solved, and the accurate recognition of objects in the image is achieved. Improve the credibility of the identification results.
Patent Information
- Application Number
- CN202110478251.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-04-29
AI Technical Summary
In the prior art, it is difficult to accurately identify different kinds of organisms or non-organisms in nature in image recognition, such as birds, dogs, flowers, cars, aircraft, etc., especially in fine-grained image analysis, the recognition accuracy is insufficient, which affects industry. and the effects of academic applications.
By obtaining the target feature vectors of the image to be processed, combining the reference feature vectors in the database, using distance parameters and setting thresholds to filter out the matching object categories, determine the target object categories of the objects in the image, and using feature extraction networks and auxiliary training The mutual learning method of the network improves the accuracy and richness of feature extraction.
It realizes accurate identification of objects in the image to be processed, improves the credibility and accuracy of the recognition results, and is suitable for fine-grained image analysis and recognition, especially in the fields of biomedical and remote sensing monitoring.
Smart Images

Figure CN113761259B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus and computer equipment. Background Art
[0002] With the development of computer vision technology, image analysis and processing have a wide range of research needs and application scenarios in both industry and academia. Image recognition has a wide range of applications, including biomedicine, remote sensing monitoring, and other fields. The corresponding recognition topics are also diverse, such as different types of birds, flowers, cars, airplanes, etc. The success of image recognition applications also depends on the accuracy of its recognition results. Accurate recognition results are of great significance to both academia and industry. Therefore, how to accurately identify the object category of an image is a problem to be solved. Summary of the invention
[0003] The embodiments of the present application provide an image processing method, apparatus, and computer device, which can accurately identify the target object category of an object in an image to be processed.
[0004] An embodiment of the present application provides an image processing method, including:
[0005] Acquire an image to be processed, perform feature extraction on the image to be processed, and obtain a target feature vector of the image to be processed;
[0006] Obtain the comparison feature vectors corresponding to M object categories from the comparison database, and determine the distance parameter between the target feature vector and each comparison feature vector, where M is a positive integer;
[0007] According to the distance parameter between the target feature vector and each comparison feature vector, N reference feature vectors are determined from the comparison feature vectors corresponding to the M object categories, where N is a positive integer;
[0008] Determine the number of target reference feature vectors in each object category corresponding to the N reference feature vectors whose distance parameter to the target feature vector is less than or equal to the set distance threshold;
[0009] If the maximum number of targets among the target numbers corresponding to the various object categories meets the set conditions, the target object category of the object in the image to be processed is determined according to the to-be-matched object category corresponding to the maximum number of targets.
[0010] An embodiment of the present application provides an image processing device, including:
[0011] An acquisition module, used for acquiring an image to be processed;
[0012] A processing module is used to perform feature extraction processing on the image to be processed to obtain a target feature vector of the image to be processed;
[0013] A determination module is used to obtain the comparison feature vectors corresponding to M object categories from the comparison database, and determine the distance parameter between the target feature vector and each comparison feature vector, where M is a positive integer;
[0014] The determination module is further used to determine N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameter between the target feature vector and each comparison feature vector, where N is a positive integer;
[0015] The determination module is further used to determine the number of target reference feature vectors whose distance parameter with the target feature vector is less than or equal to the set distance threshold in each object category corresponding to the N reference feature vectors;
[0016] The determination module is also used to determine the target object category of the object in the image to be processed according to the to-be-matched object category corresponding to the maximum target number if the maximum target number among the target numbers corresponding to each object category meets the set conditions.
[0017] On the one hand, an embodiment of the present application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program codes, and the processor is used to call program codes to execute the image processing method in the embodiment of the present application.
[0018] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the image processing method in the embodiment of the present application is executed.
[0019] Accordingly, the embodiment of the present application provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image processing method provided in one aspect of the embodiment of the present application.
[0020] In the embodiment of the present application, the reference feature vectors that meet the conditions are initially screened out through the distance parameter between the target feature vector of the image to be processed and the reference feature vector obtained from the comparison database, and then the reference feature vector that best matches the target feature vector is selected according to the target number of the reference feature vectors included in the object category, and the object category that meets the conditions is used as the target object category of the object in the image to be processed. By using this method, the distance parameter and the target number of the reference feature vector are combined to determine the object category, and the object category of the object in the image to be processed can be accurately identified. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the implementation examples or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;
[0023] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;
[0024] Figure 3a This is a schematic diagram of an interface for displaying category information of plant images provided in an embodiment of the present application;
[0025] Figure 3b This is a schematic diagram of an interface for displaying category information of multiple target objects provided by an embodiment of the present application;
[0026] Figure 4 It is a flowchart of an image processing method provided in an embodiment of the present application;
[0027] Figure 5 It is a schematic diagram of a framework based on mutual learning of dual-stream networks provided in an embodiment of the present application;
[0028] Figure 6 It is a flowchart of a method for acquiring a sample image set provided in an embodiment of the present application;
[0029] Figure 7 It is a schematic diagram of a process of cleaning plant image data provided by an embodiment of the present application;
[0030] Figure 8 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;
[0031] Fig. 9It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0033] The following is an explanation of the terms that may be involved in this application.
[0034] Image retrieval: The full name of image retrieval in English is Image Retrieval. Image retrieval aims to find images containing the same instance from the database images given a query image containing a specific instance (such as a specific target, scene, building, etc.).
[0035] Fine-grained image recognition: more refined image classification, determining more refined object classification names.
[0036] The difficulty of fine-grained image analysis compared to general image analysis tasks is that the granularity of the image category is finer, and the differences in fine-grained objects are only reflected in subtle details. At present, there is extensive research on the recognition of different kinds of biological or non-biological objects in nature, such as birds, dogs, flowers, cars, airplanes, etc., and they play different roles in different fields. The application demand for fine-grained image recognition of different categories is also very wide. For example, in ecological protection, effective identification of different kinds of organisms is an important prerequisite for ecological research.
[0037] The solution provided by the embodiment of the present application is an image processing method that combines image classification and image retrieval, which can be applied to fine-grained image analysis. The main implementation method is as follows: extract features from the acquired image to be processed to obtain the corresponding target feature vector, and obtain the target object category of the object in the image to be processed from the comparison database according to the target feature vector according to a certain matching strategy, wherein the target feature vector can be extracted using a related model of image classification, and the matching strategy can specifically be to first select a fixed number of reference feature vectors based on the distance parameter between the target feature vector and each comparison feature vector in the comparison database, and then determine the reference feature vector that meets the preset conditions, and use it as a reference feature vector that matches the target feature vector, and then obtain the target object category.
[0038] The solution provided in the embodiment of the present application belongs to the computer vision technology (Computer Vision, CV) and machine learning (Machine Learning, ML) under the field of artificial intelligence.
[0039] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0040] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify, follow and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0041] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. For example, the target feature vector of the image to be processed can be extracted using a deep neural network.
[0042] See also Figure 1, Figure 1 1 is a schematic diagram of the architecture of an image processing system provided by an embodiment of the present application, and the image processing system mainly includes an image processing device 100 and a terminal device 101. The image processing device 100 may include a comparison database, and the comparison data included in the comparison database may be used to retrieve the same or similar images. Of course, the comparison database may also be independent of the image processing device 100. A processing strategy may be pre-set on the image processing device 100, and the image sent by the terminal device 101 may be processed according to the processing strategy, and may be specifically identified to obtain the object category corresponding to the object in the image. The image processing device 100 receives the business data sent by the terminal device 101, which is specifically image data, or image data sent by other devices, and processes the received business data according to the corresponding processing strategy to obtain the processing result, and returns it to the terminal device 101. The terminal device 101 may collect and upload image data, such as image data collected by a camera, and receive the processing result returned by the image processing device 100 to visually display the final processing result to the user.
[0043] The image processing device 100 and the terminal device 101 are described in detail below, wherein:
[0044] The image processing device 100 may be equipped with a related algorithm framework for processing images, and the received image data may be processed according to this related algorithm framework, specifically: firstly, the target feature vector of the image is extracted using a feature extraction network, and then a preset number of comparison feature vectors corresponding to multiple object categories are stored in the comparison database, and a preset number of comparison feature vectors are selected as reference feature vectors according to the distance between the feature vectors, and then, according to a pre-set screening rule, the reference feature vector that best matches the target feature vector of the image is screened from multiple reference feature vectors, and then the corresponding object category is determined. The image processing device 100 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, etc., without limitation here.
[0045] An application platform with an image processing function (such as an image recognition function) can be installed on the terminal device 101. The application platform calls the terminal device 101 to collect image data, and then sends the collected image data to the image processing device 100 for processing, or the collected image data can be uniformly stored in a database, and sent to the image processing device for processing by the database. After the terminal device 101 receives the processing result of the image processing device, it can be displayed in a corresponding manner, for example, the processing result is displayed in the form of a document in the above-mentioned application platform, and the displayed content includes but is not limited to the category of the object in the image and the encyclopedia information related to the object category. If the processed image is a plant image, the planting method and origin of the plant can be displayed in text, or it can be displayed through animation or related videos. The terminal device 101 can be one or more of a smart phone, a camera, a desktop computer, a tablet computer, an MP4 player, and a laptop portable computer.
[0046] See also Figure 2 , Figure 2 1 is a flowchart of an image processing method provided by an embodiment of the present application. The execution subject in this embodiment may be a computer device or a cluster composed of multiple computer devices. The computer device may be a terminal device or a server. Here, the execution subject in this embodiment is a server (such as Figure 1 The image processing device 100 shown in FIG. 1 is used as an example for description. The image processing method may at least include the following steps S201-S205:
[0047] S201, obtaining an image to be processed, performing feature extraction processing on the image to be processed, and obtaining a target feature vector of the image to be processed.
[0048] In a possible embodiment, the image to be processed can be obtained from the terminal device, or from the background database corresponding to the terminal device, and the specific acquisition method is not limited here. The image to be processed can be any image collected by the terminal device in a corresponding manner, and the image can include an object (such as green radish), or multiple objects (such as cats, tables, trees), or no object (such as a pure black image), and the specific content included in the image to be processed is not limited here. For the acquired image to be processed, a feature extraction network can be used to perform feature extraction processing on it. The feature extraction network can be trained using the corresponding object image (such as a plant image), or a classic image feature extraction method can be used for feature extraction processing, such as local binary mode, directional gradient histogram and other methods. The feature extraction processing method is not limited here, and the target feature vector of the image to be processed can be extracted from the target object (such as a plant) in the image to be processed, which is mainly used to perform a specific mathematical expression on the content included in the image to be processed.
[0049] S202, obtaining comparison feature vectors corresponding to M object categories from a comparison database, and determining a distance parameter between the target feature vector and each comparison feature vector, where M is a positive integer.
[0050] In a possible embodiment, the comparison database stores comparison feature vectors and may also store other data information. The content of the comparison database is not limited here. The comparison feature vectors obtained in the comparison database can be used as comparison data and compared with the target feature vector. Among them, each comparison feature vector corresponds to an object category, and each object category may also correspond to one or more comparison feature vectors. Different comparison feature vectors may correspond to the same object category, and the comparison feature vectors corresponding to different object categories may also be similar. For example, there are 10 object categories in the comparison feature database, and the number of comparison feature vectors is 31. The number of comparison feature vectors corresponding to each object category is {1, 2, 3, 4, 2, 6, 3, 2, 7, 1}, respectively. The first and second object categories are similar in category, so the comparison feature vectors they include are also similar. The two comparison feature vectors of the second object category are obtained in different ways due to different collection methods, but the comparison feature vectors obtained are somewhat different. The comparison feature vectors between other different object categories are different.
[0051] The comparison database can be an offline comparison database or an online comparison database. For the construction process of the offline comparison database, it can be constructed by extracting features from images stored in an offline retrieval library independent of the comparison database, that is, extracting feature vectors of images of various object categories in advance and storing them as comparison feature vectors. The feature extraction method can be the same as the feature extraction method of the image to be processed, or it can be different from the feature extraction method of the image to be processed. For the construction process of the online comparison database, the comparison feature vectors of different object categories in some offline retrieval libraries can be extracted in advance. At the same time, during the processing process, the target feature vectors of the image to be processed processed online are stored as comparison feature vectors. That is, the images processed online can be collected at any time, and the comparison data in the comparison database can be continuously expanded during processing. This also facilitates the subsequent processing of the same image to quickly determine the comparison feature vector. The number of object categories corresponding to all comparison feature vectors in the comparison database is defined as M here. For example, there are 100,000 object categories in the comparison database, and all comparison feature vectors have 150,000. These object categories can be collected from open source data or customized data from other professional institutions. In this step, all the comparison feature vectors included in the comparison database are obtained in order to calculate the distance parameter with the target feature vector. The similarity between the target feature vector and each comparison feature vector is characterized by the distance parameter. The similarity is further processed to compare images that are similar or identical to the image to be processed. The distance parameter can be expressed in the form of Euclidean distance, Mahalanobis distance, Hamming distance, Manhattan distance, cosine distance, etc., which is not limited here, and the specific calculation process is not described in detail here.
[0052] It should be noted that in the embodiments of the present application, the object category is described as plants. The degree of description of the object category is specifically the smallest granularity description of the object. For example, the category of plants is accurate to "species". Correspondingly, the data in the comparison database can be the comparison feature vectors corresponding to different types of plant images. These data can be massive data obtained based on open source plant data and plant data from research institutes. Of course, the object category can also be extended to animals, microorganisms, non-living things (such as food, cars), etc., and the subcategories under a certain category can also be used as object categories. For example, there are also a lot of species of birds among animals that can be used as object categories. There is no restriction on the object category here.
[0053] S203 , determining N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameters between the target feature vector and each comparison feature vector, where N is a positive integer.
[0054] In a possible embodiment, the distance between feature vectors is a way to measure the similarity between two feature vectors. The distance parameter here refers to the distance parameter between the target feature vector obtained by extracting features from the image to be processed in the aforementioned step and all the comparison feature vectors obtained from the comparison database. The object categories corresponding to these distance parameters are also the M object categories included in the comparison database, that is, all object categories. According to these distance parameters, N comparison feature vectors can be determined as reference feature vectors from the comparison feature vectors corresponding to the M object categories, that is, from all the comparison feature vectors included in the comparison database, where N can be greater than, equal to, or less than M. The main connection between M and N is that the object categories corresponding to the N comparison feature vectors are part or all of the M object categories.
[0055] In a possible embodiment, the specific implementation steps for determining N reference feature vectors can be as follows: first, sort the comparison feature vectors corresponding to the M object categories according to the distance parameters between the target feature vector and each comparison feature vector from small to large, and then determine the comparison feature vectors ranked in the first N positions as reference feature vectors according to the sorting result. Specifically, the distance parameters initially determined between the target feature vector and each comparison feature vector are all unordered. In order to more quickly select the comparison feature vectors that meet the conditions, all distance parameters can be sorted first. Each distance parameter corresponds to the distance between a comparison feature vector and the target feature vector. The smaller the distance parameter, the higher the similarity between the target feature vector and the comparison feature vector. Therefore, the distance parameters can be arranged in order from small to large, so that the corresponding comparison feature vectors also have an arrangement order from small to large, and then the first N comparison feature vectors from small to large can be selected as reference feature vectors according to the arrangement order. Of course, the distance parameters can also be selected from the last N comparison feature vectors as reference feature vectors according to the order from large to small.
[0056] S204, determining the number of target reference feature vectors in each object category corresponding to the N reference feature vectors, whose distance parameter to the target feature vector is less than or equal to the set distance threshold.
[0057] In a possible embodiment, there are N distance parameters between N reference feature vectors and the target feature vector, but the number of object categories corresponding to the N reference feature vectors may be N or less than N. Among the N distance parameters, a distance parameter less than or equal to the set distance threshold is selected, and then the reference feature vector corresponding to the distance parameter and the object category corresponding to the reference feature vector can be obtained, so that the reference feature vector that meets the set distance threshold condition among the N reference feature vectors can be obtained, and then the number of reference feature vectors of the object category corresponding to these reference feature vectors that meet the set distance threshold condition can be obtained, and the number is used as the target number corresponding to each object category, and the target number can also be understood as the number of occurrences of the same object category. For example, N is 10, and there are 3 object categories corresponding to the 10 reference feature vectors, namely object a, object b, and object c. Among these 10 reference feature vectors, 8 reference feature vectors meet the corresponding distance parameter less than the set distance threshold, which are 5 reference feature vectors corresponding to object a and 3 reference feature vectors corresponding to object b. Therefore, the number of reference feature vectors 5 can be used as the number of targets corresponding to object a, and the number of reference feature vectors 3 can be used as the number of targets corresponding to object b. The process of determining the number of targets can also be specifically to first count the number of reference feature vectors included in each object category among the object categories corresponding to the N reference feature vectors, and then remove the reference feature vectors greater than the set distance threshold from these reference feature vectors, and finally obtain the number of reference feature vectors for each object category, that is, the number of targets corresponding to each object category.
[0058] S205 , if the maximum target number among the target numbers corresponding to the various object categories meets the set condition, then determine the target object category of the object in the image to be processed according to the to-be-matched object category corresponding to the maximum target number.
[0059] In a possible embodiment, the above step satisfies the reference feature vector of the set distance threshold, that is, the reference feature vector whose distance parameter between the reference feature vector and the target feature vector is less than or equal to the distance threshold, and its corresponding object category is each object category described in this step. If the maximum number of targets among the target numbers corresponding to each object category meets the preset conditions, the object category corresponding to the maximum number of targets can be regarded as the object category to be matched. At this point, it is necessary to further determine the target object category based on the object category to be matched selected from each object category.
[0060] The specific judgment method for whether the preset conditions are met can be: first determine the maximum number of targets from the target numbers corresponding to each object category, and then detect whether the maximum number of targets is greater than or equal to the set number threshold. If the maximum number of targets is greater than or equal to the set number threshold, it is determined that the maximum number of targets meets the set conditions. Among them, the set number threshold can be for N reference feature vectors, that is, if the maximum number of targets in the target numbers of N reference feature vectors reaches this set number threshold, it is considered that the maximum number of targets meets the set conditions, and the target object category of the object in the image to be processed can be further determined based on this result.
[0061] In other words, if the proportion of the maximum number of targets in the object categories corresponding to the N reference feature vectors reaches a given proportion threshold, that is, the ratio of the set number threshold and the number of object categories of the N reference feature vectors, it can be regarded as the maximum number of targets that meets the preset conditions. Among them, the proportion threshold is subsequently represented by ratio. If the maximum number of targets is less than the set number threshold, then the maximum number of targets cannot meet the set conditions, and correspondingly, the target object category of the object in the image to be processed cannot be determined according to the object category to be matched corresponding to the maximum number of targets. In this case, the default method can be used, that is, the prompt of "unable to be recognized" is returned.
[0062] According to the target object category, the information related to the target object category can be pulled out and the overall processing result can be directly returned to the terminal device, or the processing result can be stored in the database and then returned to the terminal device. The display result on the terminal device can be as follows: Figure 3a As shown, it is a schematic diagram of an interface for displaying category information of a plant image provided by an embodiment of the present application. The interface displays the category name, encyclopedia introduction, and related videos, related information or related plants of the plant image, so as to provide a variety of ways to comprehensively understand the plant.
[0063] In a possible embodiment, a specific implementation method for determining the target object category of an object in an image to be processed may also be: if the maximum number of targets among the number of targets corresponding to each object category is multiple, and the maximum number of targets satisfies a set condition, then the reference distance corresponding to each object category to be matched is determined based on a distance parameter between a reference feature vector and a target feature vector included in each object category to be matched corresponding to the maximum number of targets, and the target object category of the object in the image to be processed is determined from each object category to be matched based on the reference distance corresponding to each object category to be matched.
[0064] Since the maximum number of targets in the target number corresponding to each object category may be one or more, that is, the maximum number of targets has the same value, correspondingly, the object category to be matched corresponding to the maximum number of targets may be one or more. For example, there are three object categories in the plant image corresponding to the reference feature vector that meets the set distance threshold, and the number of targets corresponding to each object category is 2, 4, and 4 respectively. At this time, the maximum number of targets is 4, and they all meet the preset conditions, that is, greater than or equal to the set number threshold. Then there are two object categories to be matched corresponding to the maximum number of targets. Since the target object in the image to be processed usually needs to return a unique result, it is necessary to determine a target object category as the object in the image to be processed. The specific determination method adopts the specific description of this step, that is, the target object category is determined by determining the reference distance corresponding to each object category to be matched. The reference distance here can be the minimum distance parameter corresponding to the maximum number of targets, or it can be the average distance parameter of the distance parameter corresponding to the maximum number of targets, which is not limited here. According to the reference distance corresponding to each object category to be matched with the maximum number of targets, the object category to be matched corresponding to the reference distance that meets the conditions can be determined as the target object category of the object in the image to be processed.
[0065] The method described above can be used to determine the target object category of an image to be processed that includes a single object. As an extended example, when the image to be processed includes multiple objects, the target object category can be determined in the above manner for each object, so that the image to be processed can obtain multiple target object categories. In the visual display of the processing results on the terminal device, for the target object in the image to be processed, it can be displayed by segmenting the target object, matching the corresponding object category and the corresponding encyclopedia information. For details, please refer to Figure 3b , is a schematic diagram of an interface for displaying the category information of multiple target objects provided by an embodiment of the present application. It is mainly for when the image to be processed includes multiple target objects, they can be identified at the same time, and then the user selects any target object among them, such as selecting target object 1, which is a plant image, and then displaying information related to the plant image, and the related information can be the same as the aforementioned Figure 3a Similarly, it can also be adjusted according to the actual processing situation. Among them, the segmented target objects can be images of the same category, such as all plant images, or images of different categories, such as plant images and animal images. For encyclopedia information, it can be associated with the existing encyclopedia database, and then the corresponding data can be directly obtained.
[0066] For the scheme provided in this embodiment, the specific application scenario can be that a user takes a plant picture. After the plant retrieval model (including the feature extraction network), the cosine distance will be calculated with the existing retrieval features (i.e., the comparison feature vector) in the comparison database to obtain the recalled topN plant picture data with the closest distance. Traverse each picture in the topN. If the distance between the picture and the retrieval picture is less than the given threshold eps (i.e., the set distance threshold), the number of plant categories count in which the picture is located is added by one, that is, the number of targets corresponding to each object category that meets the requirements is determined, and the plant category maxcount with the largest proportion in the topN is counted, and the proportion of maxcount in N is at least greater than the given ratio. If all the above conditions are met, the encyclopedia information of the plant picture taken by the user will be returned, such as family, genus, species, light, origin and other related content. According to the online data test, the final selected N is 10, eps is 0.6, and ratio is 0.2. In addition, the specific application scenario can also be the image processing of objects such as commodities and food, which is not limited here.
[0067] In summary, the embodiments of the present application have at least the following advantages:
[0068] By setting the screening conditions of distance threshold and quantity threshold, the reference feature vectors that are consistent with the target feature vector of the image to be processed are screened out from the comparison database, the distance between the feature vectors is expanded in multiple dimensions, and the object category of the object in the image to be processed is determined from two aspects: the distance parameter and the target number of the reference feature vector. The target object category of the image to be processed can be accurately identified to ensure the credibility of the processing result.
[0069] See also Figure 4 , Figure 4 1 is a flowchart of an image processing method provided by an embodiment of the present application. The execution subject in this embodiment may be a computer device or a cluster composed of multiple computer devices. The computer device may be a terminal device or a server. Here, the execution subject in this embodiment is a server (such as Figure 1 The image processing device 100 shown in FIG. 1 is used as an example for description. The image processing method may at least include the following steps S401-S405:
[0070] S401, obtaining an image to be processed, performing feature extraction processing on the image to be processed using a feature extraction network, and obtaining a target feature vector of the image to be processed.
[0071] In a possible embodiment, the feature extraction network is trained using a sample image set in combination with an auxiliary training network. The sample image set includes multiple sample object images, each sample object image carries a category label. During the network training process, the feature extraction network and the auxiliary training network use different learning methods to perform network training based on the sample image set, and the network parameters of the feature extraction network are adjusted based on the first prediction result of the feature extraction network, the second prediction result of the auxiliary training network, and the category label.
[0072] Among them, the feature extraction network can be a residual network ResNet-152, and the auxiliary training network can be Inceptionv4. These two basic classification networks can learn from each other by using the distillation learning method. In distillation learning, there are teacher networks and student networks. The teacher network keeps the parameters unchanged during the training process and supervises and adjusts the prediction results of the student network. Based on this, mutual learning is specifically reflected in the mutual supervision and adjustment between the prediction results of the two basic classification networks. Since the ResNet-152 network has more and deeper layers and the Inception v4 network is wider, the models obtained by using the mutual learning method can absorb the advantages of these two network structures, making the feature extraction of images richer and more accurate. This is also the result of the different learning methods adopted by the feature extraction network and the auxiliary training network when learning sample object images. The feature extraction network has a deep enough network layer, better nonlinear expression ability, more complex transformation, and can fit complex features. The learning method it adopts is to abstract learning layer by layer, refine and extract knowledge, and obtain feature vectors with stronger expression ability. The auxiliary training network has a wide enough network and can increase the receptive field. The learning method it adopts is to enrich the features learned in each layer through wide channels, such as features in different directions or frequencies.
[0073] During the entire training process, the parameter quantity can be regarded as the parameter of a model, and the model only needs to be fed forward once when feeding forward. In other words, although the feature extraction network and the auxiliary training network can be adjusted simultaneously according to the prediction results and category labels, it can also be regarded as using the auxiliary training network to supervise the parameters of the feature extraction network. The feature residual network ResNet-152 is used as the feature extraction network used for prediction, and the parameter quantity is the parameter of ResNet-152. Multi-task and multi-data stream learning is introduced, and the first prediction result output by the feature extraction network and the second prediction result output by the auxiliary training network are used as the learning results of the two tasks. The parameters of the feature extraction network are adjusted in combination with the category labels of the sample object images. At the same time, the parameters of the auxiliary training network can also be adjusted to improve the generalization ability of the feature extraction network. At the same time, data enhancement can be used to increase the diversity of the training set of sample object images, such as random saturation, contrast, brightness, rotation, mirroring, random erasure, etc. to improve the robustness of the model. The problem of too few images in some categories can also be solved by balanced sampling to improve the training effect of the feature extraction network. It should be noted that after mutual learning, both ResNet-152 and Inception v4 have the ability to extract rich and accurate feature vectors. Therefore, the Inception v4 obtained through mutual learning can also be used as a feature extraction network, while the residual network ResNet-152 is regarded as an auxiliary training network. Figure 5 , is a schematic diagram of a framework based on dual-stream network mutual learning provided by an embodiment of the present application, such as Figure 5 As shown, the sample image set is input into the feature extraction network and the auxiliary training network at the same time, and the image features, that is, the feature vector corresponding to the sample object image, are obtained after processing. Then, the corresponding prediction results are obtained based on the image features, and then mutual supervised learning is performed based on the corresponding prediction results to achieve the effect of mutual learning.
[0074] In a possible embodiment, the specific training process of the feature extraction network can be referred to the following steps: first, a sample image set is obtained, and a sample object image in the sample image set is input into the feature extraction network for processing to obtain a first prediction result, and the sample object image is input into the auxiliary training network for processing to obtain a second prediction result. Then, a relative loss value is determined based on the first prediction result and the second prediction result, a first loss value is determined based on the first prediction result and the category label of the sample object image, and a second loss value is determined based on the second prediction result and the category label of the sample object image. Finally, the network parameters of the feature extraction network are adjusted based on the relative loss value, the first loss value, and the second loss value to obtain a trained feature extraction model.
[0075] Specifically, the sample image set includes multiple sample object images that carry category labels. These category labels are mathematical representations of the object categories of objects in the sample object images. For example, if the sample object image is a pine tree, the category label is marked as 01, which can be used as a real category label. The sample image set can be input into the feature extraction network and the auxiliary training network in batches, or one by one. There is no restriction on the specific input method. Since the feature extraction network and the auxiliary training network are essentially basic classification networks, the feature extraction network and the auxiliary training network are simultaneously input into any sample object image carrying a category label. The specific representation of the prediction result obtained can be similar to the representation of the category label, or it can be the probability value of the sample object image belonging to a certain category obtained after being processed by the basic classification network and then passed through the softmax layer, or other mathematical forms of representation. Among them, the softmax layer is often used in the last layer of the neural network as the output layer to perform multi-classification and map the scalar data into a probability distribution. Taking the above-mentioned first prediction result and the second prediction result as the category probability as an example, the loss value can be calculated based on the above-mentioned prediction result and the category label. The specific expression for determining the first loss value by the first prediction result and the category label of the corresponding sample object image is as follows:
[0076]
[0077] Where N represents the total number of sample object images in the sample image set, i represents the i-th sample object image, M represents the total number of object categories, m represents the m-th object category in the total number of object categories, and I(y i ,m) takes the value of 0 (when y i ≠m) or 1 (when y i =m), Represents the probability that the sample object image i is predicted as object category m by the feature extraction network.
[0078] The specific expression of the second loss value can be determined according to the second prediction result and the category label of the corresponding sample object image as follows:
[0079]
[0080] Where N represents the total number of sample object images in the sample image set, i represents the i-th sample object image, M represents the total number of object categories, m represents the m-th object category in the total number of object categories, and I(y i ,m) takes the value of 0 (when y i ≠m) or 1 (when y i =m), Represents the probability that sample image i is predicted as object category m by the auxiliary training network.
[0081] The expression for determining the relative loss value according to the first prediction result and the second prediction result is as follows:
[0082]
[0083] This expression is a representation of relative entropy (i.e. KL divergence), which specifically means the similarity between the predicted probability distributions of the two networks. The specific meanings of the relevant parameters in the expression can be found in the above L C1 and L C2 The explanation in will not be repeated here. Further, the specific expression of determining the total loss value according to the above three loss values can be as follows:
[0084] L Θ =L C1 +L C2 +D KL (p1∥p2)(4)
[0085] Among them, L C1 Represents the first loss value, L C2 Denotes the second loss value, D KL (p1∥p2) represents the relative loss value. As an extension, the above total loss value can also be assigned different weight parameters for weighted summation to adjust the influence of different loss values on the parameters of the feature extraction model. According to the total loss value, the parameters of the feature extraction network can be adjusted, and the parameters of the auxiliary training network can also be adjusted.
[0086] In the specific stage of extracting features of the image to be processed, the size of the image to be processed needs to be adjusted to 299x299, and after passing through the ResNet152 network and taking global pooling, a 1024-dimensional feature vector is obtained, which is used as the image features stored offline and the features of the online user request graph to match the feature vectors in the comparison database.
[0087] S402, obtaining comparison feature vectors corresponding to M object categories from a comparison database, and determining distance parameters between the target feature vector and each comparison feature vector, where M is a positive integer.
[0088] In a possible embodiment, similar to the method of extracting the target feature vector of the image to be processed, the comparison feature vectors corresponding to the M object categories included in the comparison database can be the feature vector extracted by the feature extraction network from the image data in the offline retrieval library, and the extracted feature vector is stored in the comparison database as the comparison data. When processing the image to be processed, the comparison feature vector in the comparison database is obtained, and the distance between the comparison feature vector and the target feature vector is calculated. The specific implementation method can be seen in Figure 2The corresponding content of step S202 in the corresponding embodiment.
[0089] S403 , determining N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameters between the target feature vector and each comparison feature vector, where N is a positive integer.
[0090] S404, determining the number of target reference feature vectors in each object category corresponding to the N reference feature vectors, whose distance parameter to the target feature vector is less than or equal to the set distance threshold.
[0091] The contents of the above steps can be found in Figure 2 Steps S203 - S204 in the method embodiment shown are not described in detail here.
[0092] S405: If the maximum target number among the target numbers corresponding to the various object categories meets the set condition, the target object category of the object in the image to be processed is determined according to the to-be-matched object category corresponding to the maximum target number.
[0093] In a possible embodiment, based on the above steps, the corresponding scheme of this embodiment can be applied to fine-grained plant image recognition, combining the advantages of fine-grained image retrieval with the disadvantages of fine-grained image classification, which can be specifically reflected in the fast speed of fine-grained image classification, the relatively light backend service construction, and only the uploaded pictures need to be classified and recognized, but the scalability is relatively poor. If a new plant category is added, the model needs to be retrained, while the scalability of fine-grained image retrieval is relatively good, and the new category can also be recognized more robustly. In the specific implementation, open source plant image data is used to clean and integrate data, adaptively adjust the algorithm, optimize the model function, and improve the accuracy of feature vector extraction; in terms of data, a fine-grained plant training database of more than 2w categories and an online plant encyclopedia retrieval library of 3.6w categories can be created, which can support the encyclopedia needs of common plants in China; in terms of performance, the recognition accuracy on the sorted evaluation set reaches about 70%, which is equal to or higher than the recognition accuracy of other products with the same function, and the common 2k plants can be recognized more accurately. At the same time, this fine-grained plant image recognition function can be embedded in some social products or content interaction platforms, for example, to achieve plant recognition by scanning. In some product experiments, the average number of background function calls per day can reach 600,000, with excellent performance in both recognition accuracy and product experience. For the rest of this step, see Figure 2 Step S205 in the method embodiment shown is not described in detail here.
[0094] In summary, the embodiments of the present application have at least the following advantages:
[0095] The target feature vector of the image to be processed is extracted by using a feature extraction network, wherein the feature extraction network is trained by combining with an auxiliary training network. In the specific processing process, the robustness of the network can be improved by preprocessing the sample image set, such as data enhancement processing. The sample object images included in the sample image set are input into the above two basic classification networks, and the prediction results are obtained to calculate the loss value to adjust the network parameters. This can help the feature extraction network to absorb the advantages of the auxiliary training network in extracting feature vectors, realize the complementary advantages of the two basic classification networks, and improve the accuracy and richness of the target feature vector. The above method is used to obtain relevant data in the comparison database, and the accuracy of the target object category in the processed image can be further guaranteed.
[0096] See also Figure 6 , is a flow chart of a method for obtaining a sample image set provided by an embodiment of the present application. The execution subject in the embodiment may be a computer device or a cluster of multiple computer devices. The computer device may be a terminal device or a server. Here, the execution subject in the embodiment is a server as an example for explanation. The method for obtaining a sample image set may at least include the following steps S601-S605:
[0097] S601, obtaining an initial object image set.
[0098] In a possible embodiment, the initial object image set may include noise images, i.e., images that do not belong to the main category of the object or unclear images. For example, if the initial object image set is a set of plant images, but includes animal images, then the animal images will be regarded as noise images. The initial object image set can be obtained from an open source image database, or from an image database constructed by a professional research institute. Usually, the initial object image set is a massive amount of image data, and the object categories involved can reach tens of thousands. The obtained initial object images are generally not directly given to the model training as sample object images, but need to be processed to ensure the effect of the training. The specific processing method is described in the following steps.
[0099] S602 , clustering the feature vectors of the initial sample object images in the initial object image set according to the initial clustering threshold to obtain one or more initial image clusters, and obtain an initial category label for each initial image cluster.
[0100] In a possible embodiment, the initial category label is used to identify the initial sample object image corresponding to the feature vector in the corresponding initial image cluster. The initial object image set can be deduplicated first, and then the corresponding feature vector is extracted from the deduplicated initial object image set using a pre-trained feature extraction network (such as an ImageNet network), and then the initial object image set is clustered using a corresponding clustering algorithm to achieve preprocessing of the object image. Specifically, the corresponding feature vector is clustered by a pre-set initial clustering threshold to obtain one or more initial image clusters, each of which is a collection of similar feature vectors. In this embodiment, the clustering algorithm uses a density-based clustering method with noise (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) as an example for explanation. The algorithm divides areas with sufficient density into clusters and finds clusters of arbitrary shapes in a spatial database with noise. It defines a cluster as the maximum set of density-connected points. Correspondingly, the initial clustering thresholds include the distance threshold esp and the density threshold min-simple. According to esp and min-simple, the core points, boundary points and noise points in the feature vector of the initial object image set can be determined, and the core points that can be reached by density can be clustered into an image cluster. Then, for each initial image cluster, an initial category label is added as the result of the first clustering, so that each initial category label can be associated with the initial sample object image to obtain the initial sample object image with the initial category label. It should be noted that other methods can also be used to cluster the initial object image set, such as K-means clustering, hierarchical clustering, system clustering, etc., which are not limited here.
[0101] S603: Update the initial clustering threshold according to the feature vectors in one or more initial image clusters.
[0102] In a possible embodiment, the initial clustering threshold is usually a large value, and correspondingly, the distance threshold and density threshold of DBSCAN are also large values. The initial image cluster clustered according to this value is a rough clustering result, so it is necessary to continuously update the clustering threshold according to the clustering result, and iterate the clustering to obtain a more accurate clustering result. The specific method can be to select a certain feature vector in any initial image cluster, and then calculate the distance between the feature vector and all other feature vectors, sort the calculated distances from small to large, find the distance value with the largest mutation, and use the distance before the distance value as the new clustering threshold and replace the initial clustering threshold with it, or the average value of all distances before the distance value can be used as the new clustering threshold to replace the initial clustering threshold. For example, if the distances between a selected feature vector and other feature vectors are ranked as [0.1, 0.9, 0.12, 0.9, 0.14, 0.3, 0.32], it can be found that 0.3 is the point with the largest distance mutation. Therefore, the previous distance 0.14 can be used as the new clustering threshold. Of course, all distances before 0.3 [0.1, 0.9, 0.12, 0.9, 0.14] can also be averaged to obtain an average value as the new clustering threshold. In addition to the above method, one or more initial image clusters can be visualized on a web page to display the clustering results, and one or both of the distance threshold or density threshold can be adjusted according to artificial experience values. The initial clustering threshold can also be adjusted decreasingly or increasing according to a set fixed step size (such as 0.05).
[0103] S604: Eliminate the initial sample object images whose feature vectors are not included in the initial image cluster in the initial object image set to obtain an intermediate object image set.
[0104] In a possible embodiment, the feature vectors of the initial sample object images included in the initial object image set are clustered, and the feature vectors included in all the obtained image clusters may not cover the feature vectors of all the initial object images. In this case, the feature vectors outside the image clusters are noise points (i.e., feature vectors of noise images), which do not belong to any image cluster. Therefore, the initial sample object images corresponding to the noise points can be removed from the initial object image set, thereby automatically cleaning the noise images in the initial object image set. The final initial sample object image set is the intermediate object image set. This method can avoid manual cleaning and denoising, saving the cost of data cleaning.
[0105] S605 , determining a sample image set according to the updated initial clustering threshold and the intermediate object image set.
[0106] In a possible embodiment, the intermediate object image set is an initial sample object image set with an initial category label, which may also include noise sample object images that have not been removed, so clustering processing is still required. A feature update model can be trained using the intermediate object image set to update the feature vector. The feature update model can be an initialized residual network, and the updated feature vector and the updated initial clustering threshold are used to perform clustering and denoising again. This process is continuously repeated, and the initial sample image of the final clustering result and the corresponding category label are associated as the sample image set.
[0107] The specific implementation method can be as follows: First, according to the initial category label of the initial sample object image in the intermediate object image set, the feature vector of the initial sample object image in the intermediate object image set is adjusted. The initial category label of the initial sample object image included in the intermediate object image set is used as the supervisory information of the prediction result of the feature update model, and the initial sample object image of the intermediate object image set is processed. Since the feature update model uses an initialized residual network, and the initial category label is used as the supervisory information to adjust the initial residual network parameters, it is different from the feature vector extracted by the feature extraction network (such as the ImageNet network) in the initial clustering. For example, the clustering threshold remains unchanged, and 10 category labels can be clustered according to the feature vector before adjustment, and 3 category labels can be clustered according to the feature vector after adjustment, indicating that the feature vector has been updated after passing through the residual network.
[0108] Then, according to the updated initial clustering threshold, the adjusted feature vectors of the initial sample object images in the intermediate object image set are re-clustered to obtain one or more target image clusters, and the target category labels of each target image cluster are obtained. Since the clustering thresholds corresponding to different categories are different, the corresponding adjusted initial clustering thresholds are also different. The adjusted feature vectors are re-clustered according to the adjusted initial clustering threshold, and the obtained target image clusters and target category labels will be different from the initial image clusters and initial category labels. In this way, more accurate clustering results and more accurate labels can be obtained through continuous iterations.
[0109] Next, according to the target category labels of each target image cluster, the category labels of the initial sample object images corresponding to the feature vectors in the corresponding target image clusters are adjusted. The target category labels of each target image cluster obtained after clustering correspond one-to-one to the initial sample object images corresponding to the feature vectors in the target image clusters. For the initial sample object images corresponding to the feature vectors in the target image clusters, their category labels may be different from the initial category labels obtained by the first clustering process. Therefore, the initial category labels will be replaced by the obtained target category labels to achieve the operation of adjusting the category labels of the initial sample object images. For example, the initial category label obtained by the first clustering of the initial sample object image A corresponding to the feature vectors in the target image cluster is 001, and the target category label obtained by the second clustering is 002. At this time, the initial category label 001 will be replaced by the target category label 002 as the current accurate category label.
[0110] Finally, when the target clustering threshold obtained by updating the feature vectors in one or more target image clusters is detected to meet the screening stop condition, the initial sample object images whose feature vectors are not included in the target image cluster in the intermediate object image set are eliminated to obtain a sample image set. After each clustering, the target clustering threshold can be adaptively adjusted according to the clustering result of the target image cluster, and the appropriate clustering effect is obtained repeatedly. For example, all feature vectors in the visualized clustering result are assigned to the corresponding image cluster, there are no isolated feature vectors (i.e., noise points) or the number of noise points is within an acceptable range, and the target clustering threshold obtained in this way is within the error range compared to the previous clustering threshold before the update, so it is determined that it meets the screening stop condition. At this time, similar to the aforementioned step S604, the initial sample object images not included in the target image in the clustering result will be eliminated, and the initial sample object images (i.e., the initial sample object images corresponding to the target cluster cluster) finally retained can be associated with the target category label as a sample image set.
[0111] The sample image set acquisition solution provided in the embodiment of the present application can also be regarded as a data cleaning method. Figure 7, is a flow chart of plant image data cleaning provided by an embodiment of the present application, mainly including: input plant data, first deduplicate the data, then perform clustering operations within the same genus according to the feature vector, that is, cluster each plant, remove noise images, and then use the web page to visualize the clustering results, adjust the threshold according to the visualized clustering results, and then automatically generate labels, input the plant data carrying the labels into the same genus retrieval model training (i.e. the aforementioned feature update model), perform feature update, and then perform clustering and denoising according to the updated features and the adjusted threshold, repeat the steps after clustering, and finally, if the threshold adjusted by the clustering result no longer affects the clustering result, clustering can be stopped, cluster cleaning is completed, and the cleaned image data is used as a sample image set. The above method is adopted because there are many types of plants, and there are many noise images of each type. If manual cleaning and denoising requires people with relevant knowledge of biology and botany, considering the time cost and labor cost, relying on algorithm clustering and denoising is more advantageous than the method based on manual data cleaning.
[0112] It should be noted that the execution subject server of this embodiment can be the same as the server of the aforementioned method embodiment, or it can be different. The server processes the sample image set obtained by the independent server and then sends it to the feature extraction network for training. Similarly, the training of the feature extraction network and the use of the trained feature extraction network can also be carried out in different servers and executed by different servers, or in the same server, which is not limited here.
[0113] In summary, the embodiments of the present application have at least the following advantages:
[0114] Through the corresponding clustering algorithm, automatic denoising and labeling are achieved for the sample object images used in the training process, especially the images involving professional fields, such as the categories to which the images of plants, animals and other related objects belong, thus avoiding the need for professional personnel to perform labeling. This can save data cleaning costs and improve the efficiency of acquiring sample image sets.
[0115] See also Figure 8 , Figure 8 : is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The above-mentioned image processing device may be a computer program (including program code) running in a computer device, for example, the image processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Figure 8 As shown, the image processing device 80 may include: an acquisition module 801, a processing module 802, and a determination module 803, wherein:
[0116] An acquisition module 801 is used to acquire an image to be processed;
[0117] The processing module 802 is used to perform feature extraction processing on the image to be processed to obtain a target feature vector of the image to be processed;
[0118] The determination module 803 is used to obtain the comparison feature vectors corresponding to M object categories from the comparison database, and determine the distance parameter between the target feature vector and each comparison feature vector, where M is a positive integer;
[0119] The determination module 803 is further used to determine N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameter between the target feature vector and each comparison feature vector, where N is a positive integer;
[0120] The determination module 803 is further used to determine the number of target reference feature vectors whose distance parameter with the target feature vector is less than or equal to the set distance threshold in each object category corresponding to the N reference feature vectors;
[0121] The determination module 803 is further configured to determine the target object category of the object in the image to be processed according to the to-be-matched object category corresponding to the maximum target number if the maximum target number among the target numbers corresponding to each object category meets the set condition.
[0122] In one embodiment, the determination module 803 is further used to sort the comparison feature vectors corresponding to the M object categories in the order of the distance parameters between the target feature vector and each comparison feature vector from small to large; and determine the comparison feature vectors ranked in the top N positions as reference feature vectors according to the sorting result.
[0123] In one embodiment, the image processing device 80 further includes: a detection module 804, wherein:
[0124] A determination module 803 is used to determine the maximum number of targets from the number of targets corresponding to each object category;
[0125] A detection module 804 is used to detect whether the maximum target quantity is greater than or equal to a set quantity threshold;
[0126] The determination module 803 is further configured to determine whether the maximum target quantity meets a set condition if the maximum target quantity is greater than or equal to a set quantity threshold.
[0127] In one embodiment, the determination module 803 is also used to determine the reference distance corresponding to each object category to be matched according to the distance parameter between the reference feature vector and the target feature vector included in each object category to be matched corresponding to the maximum target number if the maximum target number among the target numbers corresponding to each object category is multiple and the maximum target number meets the set conditions; and determine the target object category of the object in the image to be processed from each object category to be matched according to the reference distance corresponding to each object category to be matched.
[0128] In one embodiment, the processing module 802 is also used to perform feature extraction processing on the image to be processed using a feature extraction network to obtain a target feature vector of the image to be processed; wherein the feature extraction network is obtained by training using a sample image set in combination with an auxiliary training network, and the sample image set includes multiple sample object images, each sample object image carries a category label. During the network training process, the feature extraction network and the auxiliary training network use different learning methods to perform network training based on the sample image set, and the network parameters of the feature extraction network are adjusted based on the first prediction result of the feature extraction network, the second prediction result of the auxiliary training network, and the category label.
[0129] In one embodiment, the image processing device 80 further includes: an adjustment module 805, wherein:
[0130] An acquisition module 801 is used to acquire a sample image set;
[0131] A processing module 802 is used to input the sample object image in the sample image set into the feature extraction network for processing to obtain a first prediction result, and input the sample object image into the auxiliary training network for processing to obtain a second prediction result;
[0132] A determination module 803 is used to determine a relative loss value according to the first prediction result and the second prediction result, determine a first loss value according to the first prediction result and the category label of the sample object image, and determine a second loss value according to the second prediction result and the object category label of the sample object image;
[0133] The adjustment module 805 is used to adjust the network parameters of the feature extraction network based on the relative loss value, the first loss value and the second loss value to obtain a trained feature extraction model.
[0134] In one embodiment, the acquisition module 801 is also used to acquire an initial object image set, cluster the feature vectors of initial sample object images in the initial object image set according to an initial clustering threshold to obtain one or more initial image clusters, and obtain an initial category label for each initial image cluster, where the initial category label is used to identify the initial sample object image corresponding to the feature vector in the corresponding initial image cluster; update the initial clustering threshold according to the feature vectors in one or more initial image clusters; eliminate the initial sample object images in the initial object image set whose feature vectors are not included in the initial image cluster to obtain an intermediate object image set; and determine the sample image set according to the updated initial clustering threshold and the intermediate object image set.
[0135] In one embodiment, the determination module 803 is further used to adjust the feature vectors of the initial sample object images in the intermediate object image set according to the initial category labels of the initial sample object images in the intermediate object image set; re-cluster the adjusted feature vectors of the initial sample object images in the intermediate object image set according to the updated initial clustering threshold to obtain one or more target image clusters, and obtain the target category labels of each target image cluster; adjust the category labels of the initial sample object images corresponding to the feature vectors in the corresponding target image clusters according to the target category labels of each target image cluster; when the target clustering threshold obtained by updating the feature vectors in one or more target image clusters detects that the screening stop condition is met, the initial sample object images in the intermediate object image set whose feature vectors are not included in the target image clusters are eliminated to obtain a sample image set.
[0136] It is understandable that the functions of the functional modules of the image device described in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment, and the specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.
[0137] See also Fig. 9 , is a structural diagram of a computer device provided in an embodiment of the present application, the computer device 90 may include a processor 901, a memory 902, a network interface 903 and at least one communication bus 904. Among them, the processor 901 is used to schedule computer programs, and may include a central processing unit, a controller, and a microprocessor; the memory 902 is used to store computer programs, and may include a high-speed random access memory RAM, a non-volatile memory, such as a disk storage device, a flash memory device; the network interface 903 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), provide data communication functions, and the communication bus 904 is responsible for connecting various communication elements. The computer device 90 may correspond to the image processing device 100 in the previous text.
[0138] The processor 901 may be used to call a computer program in the memory to perform the following operations:
[0139] Acquire an image to be processed, perform feature extraction processing on the image to be processed, and obtain a target feature vector of the image to be processed; obtain comparison feature vectors corresponding to M object categories from a comparison database, determine the distance parameter between the target feature vector and each comparison feature vector, M is a positive integer; determine N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameter between the target feature vector and each comparison feature vector, N is a positive integer; determine the number of targets of the reference feature vectors whose distance parameter with the target feature vector is less than or equal to the set distance threshold in each object category corresponding to the N reference feature vectors; if the maximum number of targets among the number of targets corresponding to each object category meets the set condition, determine the target object category of the object in the image to be processed according to the object category to be matched corresponding to the maximum number of targets.
[0140] In one embodiment, the processor 901 is specifically used to: sort the comparison feature vectors corresponding to M object categories in order of distance parameters between the target feature vector and each comparison feature vector from small to large; and determine the comparison feature vectors ranked in the top N positions as reference feature vectors according to the sorting result.
[0141] In one embodiment, the processor 901 is also used to: determine the maximum number of targets from the number of targets corresponding to each object category; detect whether the maximum number of targets is greater than or equal to a set number threshold; if the maximum number of targets is greater than or equal to the set number threshold, determine that the maximum number of targets meets the set conditions.
[0142] In one embodiment, the processor 901 is specifically used to: if the maximum number of targets among the target numbers corresponding to each object category is multiple, and the maximum number of targets meets the set conditions, then determine the reference distance corresponding to each object category to be matched according to the distance parameter between the reference feature vector and the target feature vector included in each object category to be matched corresponding to the maximum number of targets; and determine the target object category of the object in the image to be processed from each object category to be matched according to the reference distance corresponding to each object category to be matched.
[0143] In one embodiment, the processor 901 is specifically used to: use a feature extraction network to perform feature extraction processing on the image to be processed to obtain a target feature vector of the image to be processed; wherein the feature extraction network is obtained by training using a sample image set in combination with an auxiliary training network, the sample image set includes multiple sample object images, each sample object image carries a category label, and during the network training process, the feature extraction network and the auxiliary training network use different learning methods to perform network training based on the sample image set, and the network parameters of the feature extraction network are adjusted based on the first prediction result of the feature extraction network, the second prediction result of the auxiliary training network, and the category label.
[0144] In one embodiment, the processor 901 is also used to: obtain a sample image set; input a sample object image in the sample image set into a feature extraction network for processing to obtain a first prediction result, and input the sample object image into an auxiliary training network for processing to obtain a second prediction result; determine a relative loss value based on the first prediction result and the second prediction result, determine a first loss value based on the first prediction result and the category label of the sample object image, and determine a second loss value based on the second prediction result and the object category label of the sample object image; adjust the network parameters of the feature extraction network based on the relative loss value, the first loss value, and the second loss value to obtain a trained feature extraction model.
[0145] In one embodiment, processor 901 is specifically used to: obtain an initial object image set, cluster feature vectors of initial sample object images in the initial object image set according to an initial clustering threshold to obtain one or more initial image clusters, and obtain an initial category label for each initial image cluster, wherein the initial category label is used to identify the initial sample object image corresponding to the feature vector in the corresponding initial image cluster; update the initial clustering threshold according to the feature vectors in one or more initial image clusters; eliminate the initial sample object images in the initial object image set whose feature vectors are not included in the initial image cluster to obtain an intermediate object image set; and determine the sample image set according to the updated initial clustering threshold and the intermediate object image set.
[0146] In one embodiment, processor 901 is specifically used to: adjust the feature vectors of the initial sample object images in the intermediate object image set according to the initial category labels of the initial sample object images in the intermediate object image set; re-cluster the adjusted feature vectors of the initial sample object images in the intermediate object image set according to the updated initial clustering threshold to obtain one or more target image clusters, and obtain the target category labels of each target image cluster; adjust the category labels of the initial sample object images corresponding to the feature vectors in the corresponding target image clusters according to the target category labels of each target image cluster; when the target clustering threshold obtained by updating the feature vectors in one or more target image clusters detects that a screening stop condition is met, the initial sample object images in the intermediate object image set whose feature vectors are not included in the target image clusters are eliminated to obtain a sample image set.
[0147] It should be understood that the computer device 90 described in the embodiment of the present application can execute the above Figure 2 or Figure 4 The description of the image processing method in the corresponding embodiment, and Figure 6 Corresponding to the description of the method for obtaining the sample image set in the corresponding embodiment, the above-mentioned Figure 8 The description of the image processing device 80 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.
[0148] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the computer device 90 for image processing mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the computer program can execute the above-mentioned image processing. Figure 2 or Figure 4 The description of the above-mentioned image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.
[0149] The computer-readable storage medium may be an internal storage unit of the image processing apparatus or the computer device provided in any of the aforementioned embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0150] On one hand, an embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in one aspect of the embodiment of the present application.
[0151] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.
[0152] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0153] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.
[0154] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. An image processing method, characterized in that: include: Acquire an image to be processed, perform feature extraction processing on the image to be processed, and obtain a target feature vector of the image to be processed; Obtaining comparison feature vectors corresponding to M object categories from a comparison database, and determining distance parameters between the target feature vector and each comparison feature vector, where M is a positive integer; Determine N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameter between the target feature vector and each comparison feature vector, where N is a positive integer; Determine the number of target reference feature vectors in each object category corresponding to the N reference feature vectors, whose distance parameter from the target feature vector is less than or equal to a set distance threshold; If the maximum number of targets among the target numbers corresponding to the various object categories is one, and the maximum number of targets meets the set condition, determining the target object category of the object in the image to be processed according to the to-be-matched object category corresponding to the maximum number of targets; If the maximum number of targets among the target numbers corresponding to the various object categories is multiple, and the maximum number of targets meets the set conditions, then the reference distance corresponding to each object category to be matched is determined according to the distance parameter between the reference feature vector included in each object category to be matched corresponding to the maximum number of targets and the target feature vector, and the target object category of the object in the image to be processed is determined from the various object categories to be matched according to the reference distance corresponding to the each object category to be matched.
2. The method according to claim 1, characterized in that The determining N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameter between the target feature vector and each comparison feature vector comprises: Sorting the comparison feature vectors corresponding to the M object categories in ascending order of distance parameters between the target feature vector and each comparison feature vector; According to the sorting result, the first N comparison feature vectors are determined as reference feature vectors.
3. The method according to claim 1, characterized in that The method further comprises: Determining the maximum number of targets from the number of targets corresponding to each of the object categories; Detecting whether the maximum target quantity is greater than or equal to a set quantity threshold; If the maximum target quantity is greater than or equal to the set quantity threshold, it is determined that the maximum target quantity meets the set condition.
4. The method according to claim 1, characterized in that The step of performing feature extraction processing on the image to be processed to obtain a target feature vector of the image to be processed includes: Performing feature extraction processing on the image to be processed by using a feature extraction network to obtain a target feature vector of the image to be processed; Among them, the feature extraction network is obtained by training using a sample image set in combination with an auxiliary training network. The sample image set includes multiple sample object images, each sample object image carries a category label. During the network training process, the feature extraction network and the auxiliary training network use different learning methods to perform network training based on the sample image set, and the network parameters of the feature extraction network are adjusted based on the first prediction result of the feature extraction network, the second prediction result of the auxiliary training network and the category label.
5. The method according to claim 4, characterized in that The method further comprises: Acquire the sample image set; Inputting a sample object image in the sample image set into the feature extraction network for processing to obtain a first prediction result, and inputting the sample object image into the auxiliary training network for processing to obtain a second prediction result; Determining a relative loss value according to the first prediction result and the second prediction result, determining a first loss value according to the first prediction result and the category label of the sample object image, and determining a second loss value according to the second prediction result and the object category label of the sample object image; Based on the relative loss value, the first loss value, and the second loss value, the network parameters of the feature extraction network are adjusted to obtain a trained feature extraction model.
6. The method according to claim 5, characterized in that The obtaining of the sample image set comprises: Acquire an initial object image set, perform clustering processing on feature vectors of initial sample object images in the initial object image set according to an initial clustering threshold, obtain one or more initial image clusters, and obtain an initial category label for each initial image cluster, wherein the initial category label is used to identify the initial sample object image corresponding to the feature vector in the corresponding initial image cluster; updating the initial clustering threshold according to feature vectors in the one or more initial image clusters; Eliminate the initial sample object images whose feature vectors are not included in the initial image cluster from the initial object image set to obtain an intermediate object image set; The sample image set is determined according to the updated initial clustering threshold and the intermediate object image set.
7. The method according to claim 6, characterized in that The step of determining the sample image set according to the updated initial clustering threshold and the intermediate object image set includes: adjusting the feature vectors of the initial sample object images in the intermediate object image set according to the initial category labels of the initial sample object images in the intermediate object image set; re-clustering the adjusted feature vectors of the initial sample object images in the intermediate object image set according to the updated initial clustering threshold to obtain one or more target image clusters, and obtain a target category label for each target image cluster; According to the target category labels of each target image cluster, adjusting the category labels of the initial sample object images corresponding to the feature vectors in the corresponding target image clusters; When the target clustering threshold updated according to the feature vectors in the one or more target image clusters detects that a screening stop condition is met, initial sample object images whose feature vectors are not included in the target image cluster in the intermediate object image set are eliminated to obtain the sample image set.
8. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A processing module, used for performing feature extraction processing on the image to be processed to obtain a target feature vector of the image to be processed; A determination module is used to obtain comparison feature vectors corresponding to M object categories from a comparison database, and determine a distance parameter between the target feature vector and each comparison feature vector, where M is a positive integer; The determination module is further used to determine N reference feature vectors from the comparison feature vectors corresponding to the M object categories according to the distance parameter between the target feature vector and each comparison feature vector, where N is a positive integer; The determination module is further used to determine the number of target reference feature vectors whose distance parameter with the target feature vector is less than or equal to a set distance threshold in each object category corresponding to the N reference feature vectors; The determination module is further configured to determine the target object category of the object in the image to be processed according to the to-be-matched object category corresponding to the maximum number of targets if the maximum number of targets among the numbers of targets corresponding to the various object categories is one and the maximum number of targets meets a set condition; The determination module is further used to determine the reference distance corresponding to each object category to be matched according to the distance parameter between the reference feature vector included in each object category to be matched corresponding to the maximum target number and the target feature vector if the maximum target number among the target numbers corresponding to each object category is multiple and the maximum target number meets the set conditions, and determine the target object category of the object in the image to be processed from the each object category to be matched according to the reference distance corresponding to the each object category to be matched.
9. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code to execute the image processing method described in any one of claims 1-7.
10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to perform the image processing method according to any one of claims 1 to 7.
