Image processing method and related device

Abstract

The invention provides an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: encoding a first image to obtain a first image feature; performing first identification processing on the first image feature to obtain a first identification result; based on the first forward label library and the first identification result, performing second identification processing on the first image feature to obtain a second identification result; based on a negative label library and the first identification result, performing third identification processing on the first image feature to obtain a third identification result; and fusing the first recognition result, the second recognition result and the third recognition result to obtain a fourth recognition result of the first image, so that the image processing accuracy can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an image processing method and related devices. Background Art

[0002] At the forefront of today's science and technology, visual language models are booming, achieving image classification through shared embedding spaces obtained from large-scale image-text pairs. However, when faced with a variety of specific downstream image processing tasks, there are significant differences and offsets in the image domain and probability distribution of downstream images. Image classification faces huge obstacles in practical application scenarios and has low computational efficiency. When faced with new and unseen data, it often performs poorly and cannot be effectively migrated and applied. Summary of the invention

[0003] The embodiments of the present application provide an image processing method and related devices, which can improve the accuracy of image processing.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides an image processing method, the method comprising:

[0006] Encoding the first image to obtain a first image feature;

[0007] Performing a first recognition process on the first image feature to obtain a first recognition result;

[0008] Based on the first positive label library and the first recognition result, performing a second recognition process on the first image feature to obtain a second recognition result, wherein the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than a first confidence threshold;

[0009] Based on the negative label library and the first recognition result, performing a third recognition process on the first image feature to obtain a third recognition result, wherein the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than a second confidence threshold;

[0010] The first recognition result, the second recognition result, and the third recognition result are integrated to obtain a fourth recognition result of the first image.

[0011] The present application provides an image processing device, including:

[0012] A preprocessing module, used for encoding the first image to obtain a first image feature;

[0013] a recognition processing module, configured to perform a first recognition processing on the first image feature to obtain a first recognition result; based on a first positive label library and the first recognition result, perform a second recognition processing on the first image feature to obtain a second recognition result, wherein the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than a first confidence threshold; based on a negative label library and the first recognition result, perform a third recognition processing on the first image feature to obtain a third recognition result, wherein the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than a second confidence threshold;

[0014] The result fusion module is used to fuse the first recognition result, the second recognition result and the third recognition result to obtain a fourth recognition result of the first image.

[0015] An embodiment of the present application provides an electronic device, including:

[0016] A memory for storing computer executable instructions;

[0017] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the image processing method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the image processing method provided by the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] Based on the first positive label library and the first recognition result, a second recognition process is performed on the first image feature to obtain a second recognition result. Based on the negative label library and the first recognition result, a third recognition process is performed on the first image feature to obtain a third recognition result. Since the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than the first confidence threshold, and the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than the second confidence threshold, the first positive label library includes image features that are easy to obtain correct recognition results, and the negative label library includes image features that are easy to obtain incorrect recognition results. By using the image features included in the first positive label library and the image features included in the negative label library, the first image feature is recognized and processed, and the correct image features and the incorrect image features can be combined to assist the recognition process of the first image feature, thereby improving the generalization of image recognition. At the same time, the recognition results from different sources are integrated to improve the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 3A is a first flow chart of the image processing method provided in an embodiment of the present application;

[0025] Figure 3B is a second flow chart of the image processing method provided in an embodiment of the present application;

[0026] Figure 3C is a third flow chart of the image processing method provided in an embodiment of the present application;

[0027] Figure 3D is a fourth flow chart of the image processing method provided in an embodiment of the present application;

[0028] Figure 3E is a fifth flow chart of the image processing method provided in the embodiment of the present application;

[0029] Figure 3F is a sixth flow chart of the image processing method provided in an embodiment of the present application;

[0030] Figure 3G is a seventh flow chart of the image processing method provided in an embodiment of the present application;

[0031] Figure 3H is an eighth flow chart of the image processing method provided in an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of the image recognition process provided in an embodiment of the present application.

[0033] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0039] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) Recognition processing, which is used to identify objects, scenes, or activities in the image to determine the category to which the image belongs.

[0042] 2) A forward label library, including multiple forward labels, each of which includes an image feature of a historical image and a recognition result of the historical image, and a confidence level of the recognition result included in the forward label is greater than a first confidence threshold.

[0043] 3) A negative label library, including multiple negative labels, each negative label includes image features of historical images and recognition results of historical images, and the confidence of the recognition results included in the negative labels is less than a second confidence threshold, and the second confidence threshold is less than the first confidence threshold.

[0044] In the related art, the probability distribution of the image is obtained only by calculating the dot product of the image features and the text features, which reduces the accuracy of image classification. To address the above problem, the embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of image processing.

[0045] The image processing method described in the embodiments of the present application can be applied to various fields, for example, anomaly detection, image classification and multiple categories of fields, that is, the image processing method in the embodiments of the present application is not limited to a certain field.

[0046] The following describes an exemplary application of the electronic device provided in the embodiment of the present application. The device provided in the embodiment of the present application can be implemented as a terminal or a server. The following describes an exemplary application when the device is implemented as a server.

[0047] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the image processing system 100 provided in an embodiment of the present application. To support an image processing application, a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0048] The terminal 400 is used to send the first image and the label text of the first image to the server 200 through the network 300, and the server 200 is used to encode the first image to obtain the first image feature; perform a first recognition process on the first image feature to obtain a first recognition result; based on the first positive label library and the first recognition result, perform a second recognition process on the first image feature to obtain a second recognition result, wherein the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than the first confidence threshold; based on the negative label library and the first recognition result, perform a third recognition process on the first image feature to obtain a third recognition result, wherein the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than the second confidence threshold; fuse the first recognition result, the second recognition result and the third recognition result to obtain a fourth recognition result of the first image, return the fourth recognition result to the terminal 400, and the terminal 400 displays the fourth recognition result through the graphical interface 410.

[0049] Next, an example of image processing performed by terminal 400 will be described.

[0050] In some embodiments, terminal 400 can independently complete image processing tasks. For example, terminal 400 is used to encode a first image to obtain a first image feature; perform a first recognition process on the first image feature to obtain a first recognition result; based on the first positive label library and the first recognition result, perform a second recognition process on the first image feature to obtain a second recognition result, wherein the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than the first confidence threshold; based on the negative label library and the first recognition result, perform a third recognition process on the first image feature to obtain a third recognition result, wherein the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than the second confidence threshold; fuse the first recognition result, the second recognition result and the third recognition result to obtain a fourth recognition result of the first image, and display the fourth recognition result through the graphical interface 410.

[0051] In one implementation scenario, the server or terminal can perform anomaly detection on the abnormal image, encode the abnormal image, and obtain abnormal features; perform a first abnormal recognition on the abnormal features to obtain a first abnormal result; based on the positive label library and the first abnormal result, perform a second abnormal recognition on the abnormal features to obtain a second abnormal result, wherein the confidence of the abnormal results included in the positive label library is greater than the first confidence threshold; based on the negative label library and the first abnormal result, perform a third abnormal recognition on the abnormal features to obtain a third abnormal result, wherein the confidence of the abnormal results included in the negative label library is less than the second confidence threshold; and fuse the first abnormal result, the second abnormal result, and the third abnormal result to obtain a fourth abnormal result of the abnormal image.

[0052] In one implementation scenario, a server or a terminal may perform image classification, encode an image, and obtain image features; perform a first classification process on the image features to obtain a first category; based on a positive label library and the first category, perform a second classification process on the features to obtain a second category, wherein the confidence that the image included in the positive label library belongs to the category is greater than a first confidence threshold; based on a negative label library and the first category, perform a third classification process on the features to obtain a third category, wherein the confidence that the image included in the negative label library belongs to the category is less than a second confidence threshold; and fuse the first category, the second category, and the third category to obtain a fourth category of the image.

[0053] In some embodiments, server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0054] The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0055] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1 In the terminal 400 or server 200, the electronic device 500 includes: at least one processor 510, a memory 550, and at least one network interface 520. The various components in the server 200 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .

[0056] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0058] In some embodiments, when the terminal 400 independently completes the image processing task, the server 200 provided in the embodiment of the present application does not include the user interface 530 .

[0059] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard disk drive, an optical disk drive, etc. The memory 550 may optionally include one or more storage devices that are physically located away from the processor 510.

[0060] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0061] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0062] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0063] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0064] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0065] In some embodiments, when the image processing task is completed independently by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0066] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions; in some embodiments, when the embodiment independently completes the image processing task by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0067] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The image processing device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a pre-processing module 5551, a recognition processing module 5552, and a result fusion module 5553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0068] It should be noted that in the following image processing examples, those skilled in the art may apply the image processing method provided in the embodiments of the present application to image processing (such as image recognition) based on their understanding of the following.

[0069] See also Figure 3A , Figure 3A is a first flow chart of the image processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are used for illustration. The image processing method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will take the collaborative implementation by a server and a terminal as an example for illustration.

[0070] In step 101, a first image is encoded to obtain a first image feature.

[0071] Here, the first image is an image used for image recognition. Image processing is performed on the first image to obtain a fourth recognition result of the first image. Encoding is used to map image data (such as the first image) to a high-dimensional space, so as to convert visible image data into imperceptible feature vectors.

[0072] In some embodiments, the above-mentioned "encoding the first image to obtain first image features" can be implemented in the following ways: performing convolution processing on the first image to obtain convolution features, performing pooling processing on the convolution features to obtain pooling features, and performing mapping processing on the pooling features to obtain first image features.

[0073] For example, the first image is convolved to obtain convolution features (such as [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]), the convolution features are pooled to obtain pooled features (such as [0.14, 0.13, -0.06]), and the pooled features are mapped to obtain first image features (such as [0.52, 0.13, -0.06, 0.15].

[0074] Continue to see Figure 3A In step 102, a first recognition process is performed on the first image feature to obtain a first recognition result.

[0075] Here, the first recognition process is used to recognize an object, scene or activity in an image to determine the category to which the first image belongs, and the first recognition result is used to characterize the probability that the first image belongs to different categories.

[0076] In some embodiments, Figure 3A The illustrated step 102 may be implemented in the following manner: encoding the label text of the first image to obtain text features; and performing a first recognition process on the first image features based on the text features to obtain a first recognition result.

[0077] Here, the label text of the first image is a text constructed based on the category of the image, which is used to represent the category information of the image. The encoding is used to map text data (such as the label text of the first image) to a high-dimensional space, which is used to convert visible text data into imperceptible feature vectors.

[0078] In some embodiments, before the above “encoding the label text of the first image to obtain text features”, the following processing is performed: determining the replacement text of the prompt word template and the category of the image, replacing the replacement text of the prompt word template with the category, and obtaining the label text.

[0079] Here, the category of the image is a set of multiple categories (such as [cat, dog, tiger, mouse]), which is used to characterize the category to which the image may belong. The first image belongs to one of the multiple categories. The replacement text is text used to indicate the category to be replaced (such as *), and the prompt word template is a template text used to construct a label text (such as "This is an image about *"), so that the neural network model can learn specific features based on the category of the image.

[0080] For example, determine the replacement text (such as *) of the prompt word template (such as "This is an image that may be about *"), and determine the category (such as [cat, dog, tiger, mouse]), replace the replacement text (such as *) of the prompt word template (such as "This is an image that may be about *") with the category (such as dog), and obtain the label text (such as "This is an image that may be about cats, dogs, tigers, and mice").

[0081] Continuing from the above embodiment, the above step of “encoding the label text of the first image to obtain text features” is consistent with the above step of “encoding the first image to obtain first image features”, which will not be repeated here.

[0082] Continuing from the above embodiment, the above “based on the text feature, performing a first recognition process on the first image feature to obtain a first recognition result” can be achieved in the following way: performing a dot product process on the first image feature and the text feature to obtain a second dot product feature, and normalizing the second dot product feature to obtain a first recognition result.

[0083] Here, the dot product processing is used to calculate the sum of the multiplication of corresponding elements of two vectors, and normalization is used to adjust the data to a uniform scale to avoid deviations caused by different features having different measurement units and numerical ranges. The embodiment of the present application does not limit the normalization method, and normalization can be batch normalization, layer normalization, etc.

[0084] In some embodiments, the above “normalizing the second dot product feature to obtain the first recognition result” can be implemented in the following manner: based on a preset norm parameter, performing a power operation on each element of the second dot product feature to obtain a power element; summing each power element to obtain a sum element; performing a power operation on the sum element based on the reciprocal of the norm parameter to obtain a target element; determining the ratio of each element of the second dot product feature to the target element as the first recognition result, wherein the norm parameter (i.e., exponent) is used to characterize the number of times the element itself is multiplied, and the norm parameter is a real number greater than 0. And performing a dot product process to obtain a second dot product feature (such as 44).

[0085] For example, given a first image feature (such as [[2, 1, 1], [4, 2, 2]]), a text feature (such as [2, 4, -6]), the first image feature is dot-producted with the text feature to obtain a second dot-product feature (such as [2, 4]), a norm parameter (such as 2), the square of each element of the second dot-product feature is calculated to obtain a power element (such as 2 squared is 4, 4 squared is 16), the sum of each power element is calculated to obtain a sum element (such as 20), and the norm parameter is used. The reciprocal of (such as 0.5, i.e., square root) is used to perform a power operation on the summed elements to obtain a target element (such as 4.5), and the ratio of each element of the second dot product feature to the target element (such as the ratio of 2 to 4.5 is 0.44, and the ratio of 2 to 4.5 is 0.88) is determined as the first recognition result (such as [0.12, 0.88]), where 0.12 is used to represent the probability that the first image belongs to the first image category, and 0.88 is used to represent the probability that the first image belongs to the second image category.

[0086] Continue to see Figure 3A In step 103, based on the first positive label library and the first recognition result, a second recognition process is performed on the first image feature to obtain a second recognition result.

[0087] The confidence level of the recognition result corresponding to the second image feature included in the first positive label library is greater than the first confidence level threshold.

[0088] Here, the first forward label library includes multiple labels, each label includes image features of historical images and recognition results of historical images, the confidence of the recognition results included in the labels is greater than the first confidence threshold, the second recognition processing is a recognition processing method different from the first recognition processing, and is used to identify objects, scenes or activities in the image, so as to easily determine the correct category to which the first image belongs, and the second recognition result is used to characterize the probability that the first image belongs to different categories, wherein the probability that the first image belongs to the correct category has a high proportion, and the recognition result corresponding to the second image feature is obtained by performing the first recognition processing on the second image feature based on the text feature of the image corresponding to the second image feature, and the confidence is a measure of the certainty of the model on the recognition result, which is used to indicate the degree of certainty of the model for each category it recognizes. The embodiment of the present application does not limit the confidence of the recognition result, and the confidence of the recognition result can be the entropy value of the recognition result, the variance of the recognition result, etc.

[0089] In some embodiments, see Figure 3B , Figure 3B is a second flow chart of the image processing method provided in the embodiment of the present application, in step Figure 3A Before 103 shown, execute Figure 3B Steps 201A to 204A are described in detail below.

[0090] In step 201A, the second image is encoded to obtain second image features, and the second image is an image that has been processed.

[0091] Here, the embodiment of the present application does not limit the second image, and the second image may be a historical image that has been processed.

[0092] It should be noted that step 201A is similar to step 101 and will not be described in detail here.

[0093] In step 202A, a first recognition process is performed on the second image feature to obtain a fifth recognition result.

[0094] It should be noted that step 202A is similar to step 102 and will not be described in detail here.

[0095] In step 203A, when the confidence of the fifth recognition result is greater than the first confidence threshold, the second image feature and the fifth recognition result are combined to obtain a first label for the second image.

[0096] For example, the second image features (such as [[2, 1, 1], [4, 2, 2]]) and the fifth recognition result (such as [0.12, 0.88]) are combined to obtain the first label of the second image (such as [[[2, 1, 1], [4, 2, 2]], [0.12, 0.88]]).

[0097] In step 204A, a first update process is performed on the second forward label library based on the first label to obtain a first forward label library.

[0098] Here, the second forward label library is a forward label library that has not been updated. The embodiment of the present application does not limit the second forward label library. The second forward label library can be an empty forward label library or a forward label library containing labels. The construction process of the labels contained in the second forward label library is consistent with the construction process of the first label, and will not be repeated here.

[0099] In some embodiments, see Figure 3C , Figure 3C is a third flow chart of the image processing method provided in the embodiment of the present application, for Figure 3B Step 204A shown can be performed by Figure 3C Steps 2041A to 2043A are implemented as described in detail below.

[0100] In step 2041A, a first category of the first tag is determined.

[0101] Here, the first category is the category to which the first label belongs, that is, the category to which the image corresponding to the first label belongs. Taking the category to which the image corresponding to the first label belongs as the animal category as an example, the first category of the first label is the animal category.

[0102] In some embodiments, step 2041A can be implemented in at least one of the following ways: determining the category with the highest probability in the recognition results included in the first label as the first category; or, when the probability in the recognition results included in the first label is greater than the probability threshold, determining the category corresponding to the probability as the first category.

[0103] For example, the category with the largest probability (dog corresponding to probability 0.6) in the recognition results included in the first label (such as [0.6, 0.2, 0.2], where the data 0.6 with index subscript 0 is used to indicate that the probability that the image corresponding to the recognition result belongs to the cat class is 0.6, the data 0.2 with index subscript 1 is used to indicate that the probability that the image corresponding to the recognition result belongs to the dog class is 0.2, and the data 0.2 with index subscript 2 is used to indicate that the probability that the image corresponding to the recognition result belongs to the rat class is 0.2) is determined as the first category; or, when the probability in the recognition results included in the first label is greater than the probability threshold (such as 0.5), the category corresponding to the probability (such as 0.6) (such as dog) is determined as the first category.

[0104] In step 2042A, when the number of labels belonging to the first category in the second forward label library is less than the first quantity threshold, the first label is added to the second forward label library, and the second forward label library after the first label is added is used as the first forward label library.

[0105] For example, when the number (e.g., 2) of labels belonging to the first category in the second forward label library (e.g., labels belonging to the dog category in the second forward label library include label A and label B) is less than a first quantity threshold (e.g., 5), the first label is added to the second forward label library, and the second forward label library after adding the first label is used as the first forward label library (e.g., labels belonging to the dog category in the first forward label library include the first label, label A, and label B).

[0106] In step 2043A, when the number of labels belonging to the first category in the second forward label library is greater than or equal to the first quantity threshold, the second forward label library is updated first based on the confidence of the fifth recognition result, and the second forward label library after the first update is used as the first forward label library.

[0107] In some embodiments, before the above-mentioned "performing a first update processing on the second forward label library based on the confidence of the fifth recognition result, and using the second forward label library after the first update processing as the first forward label library", the confidence of the fifth recognition result can be determined in the following way: performing the following processing for each probability of the fifth recognition result, calculating the product of the logarithm of the probability and the probability; adding the negative of each product to obtain the confidence of the fifth recognition result.

[0108] For example, the product of the logarithm (eg -0.23) of the probability (eg 0.8) and the probability (eg -0.446) is calculated; the negative of each product (eg -0.446 and -3.22) (eg 0.446 and 3.22) is added together to obtain the confidence level of the fifth recognition result (eg 3.666).

[0109] Continuing from the above embodiment, before the above “performing a first update processing on the second forward label library based on the confidence of the fifth recognition result to obtain a first forward label library”, the confidence of the fifth recognition result can also be determined in the following way: perform the following processing for each probability of the fifth recognition result, add the square of each probability, and obtain the confidence of the fifth recognition result.

[0110] For example, the squares of each probability (such as 0.8 and 0.2) (such as 0.64 and 0.04) are added together to obtain the confidence level of the fifth recognition result (such as 0.68).

[0111] In some embodiments, the first update processing of the second forward label library based on the confidence of the fifth recognition result in step 2043A and the use of the second forward label library after the first update processing as the first forward label library can be achieved in the following manner: determining the second label from the second forward label library, and determining the first confidence of the recognition result included in the second label; when the confidence of the fifth recognition result is greater than the first confidence, replacing the second label in the second forward label library with the first label, and using the second forward label library after replacing the second label as the first forward label library.

[0112] Here, the confidence of the recognition result included in the second tag is the minimum confidence of the recognition results included in the tags of the second forward tag library.

[0113] In some embodiments, the above-mentioned "determining the second label from the second forward label library" can be implemented in the following manner: calculating the confidence of the recognition results included in the labels in the second forward label library; determining the label corresponding to the minimum confidence as the second label, and the minimum confidence is the smallest confidence among multiple confidences.

[0114] For example, the confidence of the recognition results included in the labels in the second forward label library is calculated (such as the confidence of the recognition results included in label A is 2.8, and the confidence of the recognition results included in label B is 2); the label (such as label B) corresponding to the minimum confidence (such as 2) among multiple confidences is determined as the second label, and when the confidence of the fifth recognition result (2.5) is greater than the first confidence, the second label (such as label B) in the second forward label library is replaced with the first label, and the second forward label library after replacing the second label is used as the first forward label library (the first forward label library includes label A and the first label, the confidence of the recognition result included in label A is 2.8, and the confidence of the recognition result included in the first label is 2.5).

[0115] Through the embodiments of the present application, the label corresponding to the minimum confidence is replaced, and the overall confidence of the positive label library is continuously improved, thereby improving the accuracy of image recognition processing.

[0116] In some embodiments, after the above-mentioned "performing a first update processing on the second forward label library based on the confidence of the fifth recognition result to obtain a first forward label library", the following processing can also be performed to determine whether the first forward label library meets the first condition. If the first forward label library meets the first condition, a third category is determined from the categories to which the labels in the first forward label library belong, and based on the third category, a second update processing is performed on the first forward label library to obtain the first forward label library after the second update processing to replace the first forward label library.

[0117] The first condition includes one of the following: the number of tags in the first forward tag library is greater than the second number threshold; the number of updates of tags in the first forward tag library is greater than or equal to the third number threshold.

[0118] It should be noted that the above step of "determining a third category from the categories to which the labels in the first forward label library belong, and based on the third category, performing a second update process on the first forward label library to obtain the first forward label library after the second update process" is similar to step 202B, see step 202B for details.

[0119] In some embodiments, see Figure 3D , Figure 3D is a fourth flow chart of the image processing method provided in the embodiment of the present application, Figure 3A Before step 103 shown, execute Figure 3D Steps 201B to 202B are described in detail below.

[0120] In step 201B, when the first condition is met, a second category is determined from the categories to which the tags in the third forward tag library belong.

[0121] The first condition includes one of the following: the number of tags in the third forward tag library is greater than the second number threshold; the number of updates of tags in the third forward tag library is greater than or equal to the third number threshold.

[0122] Here, the embodiment of the present application does not limit the third forward tag library. The third forward tag library may be consistent with the second forward tag library or inconsistent with the second forward tag library. Step 201B is consistent with step 201A and will not be described in detail here.

[0123] It should be noted that the number of times the label is updated is the historical number of times the first update process is performed on the third forward label library. After each first update process is performed on the third forward label library, the number of times the label is updated is increased, and each time the second update process is performed on the third forward label library, the number of times the label is updated is reset.

[0124] In some embodiments, the above “determining the second category from the categories to which the labels of the third forward label library belong” can be implemented in the following manner: filtering out the second category from the categories to which the labels of the third forward label library belong.

[0125] It should be noted that the embodiment of the present application does not limit the filtering method for filtering out the second category from the category to which the label of the third forward label library belongs. The filtering method can be based on the number of labels, the confidence of the recognition results contained in the labels, and the number of times the labels in the third forward label library are updated. The category to which the labels of the third forward label library belong is filtered.

[0126] Continuing from the above embodiment, the above-mentioned "filtering out the second category from the categories to which the labels of the third forward label library belong" can be implemented by at least one of the following methods: determining the second category: based on the number of labels in the category, filtering out the second category from the categories to which the labels in the second forward label library belong; or, determining the maximum confidence from the confidences of the recognition results contained in the labels in the second forward label library, and determining the category to which the labels corresponding to the maximum confidence belong as the second category; or, based on the number of updates of the labels in the category, filtering out the second category from the categories to which the labels in the second forward label library belong.

[0127] Continuing from the above embodiment, the above “based on the number of labels in the category, filtering out the second category from the categories to which the labels in the second forward label library belong” can be implemented in the following way: when the number of labels in the category is greater than the quantity threshold, the category is determined as the second category, or, based on the number of labels in the category, the categories are sorted to obtain sorting results, and some categories in the sorting results are determined as the second category.

[0128] For example, when the number of labels in a category (such as 85) is greater than a quantity threshold (such as 80), the category is determined to be the second category; alternatively, category A corresponds to the number of labels in the category (such as 85), and category B corresponds to the number of labels in the category (such as 80); the categories are sorted in ascending order based on the number of labels in the category (such as 85 and 80) to obtain an ascending sorting result (such as [B, A]), and the latter part of the categories in the ascending sorting result (such as A) is determined to be the second category; based on the number of labels in the category (such as 85 and 80), the categories are sorted in descending order to obtain a descending sorting result (such as [A, B]), and the former part of the categories in the descending sorting result (such as A) is determined to be the second category.

[0129] Continuing with the above example, the maximum confidence (such as 0.88) is determined from the confidences of the recognition results contained in the labels in the second forward label library (such as the confidence of the recognition result contained in label 1 belonging to category A is 0.85, the confidence of the recognition result contained in label 2 belonging to category A is 0.78, and the confidence of the recognition result contained in label 3 belonging to category B is 0.88), and the category (such as category B) to which the label corresponding to the maximum confidence (such as label 3) belongs is determined as the second category.

[0130] Continuing from the above embodiment, the step of “filtering out the second category from the categories to which the labels in the second forward label library belong based on the number of updates of the labels in the category” is consistent with the step of “filtering out the second category from the categories to which the labels in the second forward label library belong based on the number of labels in the category”, and will not be repeated here.

[0131] Through the embodiments of the present application, the labels of the categories to be updated are screened according to the number of labels in the categories or the number of updates, thereby avoiding the update of all labels and improving the update efficiency of the forward label library.

[0132] In step 202B, based on the second category, a second update process is performed on the third forward label library to obtain a first forward label library.

[0133] In some embodiments, see Figure 3E , Figure 3E is a fifth flow chart of the image processing method provided in the embodiment of the present application, for Figure 3D Step 202B shown can be performed by Figure 3E Steps 2021B to 2022B are implemented as described in detail below.

[0134] In step 2021B, for each cluster of the third forward label library, a third label is determined from the cluster, where the cluster includes labels belonging to the second category.

[0135] Here, a cluster is a collection of multiple labels, and the labels within the cluster are similar. In the process of clustering the labels belonging to the second category, the labels belonging to the second category are allocated to different clusters.

[0136] In some embodiments, before determining the third label from the cluster in step 2021B, the following steps are performed to cluster each fifth label belonging to the second category in the third forward label library to obtain the subcategory to which each fifth label belongs; and the fifth labels belonging to the same subcategory form a cluster.

[0137] Here, a category is an element vector in a set that has common characteristics or attributes. The embodiment of the present application does not limit the category, and the category may be an animal category, a plant category, etc.; clustering is a method of grouping data (such as the fifth label), and the data in the group has similar characteristics. The embodiment of the present application does not limit the clustering method, and the clustering method may be K-means clustering, hierarchical clustering, etc. A subcategory is a subdivision of a category, and is a more specific grouping in a category. A subcategory inherits the characteristics of the category and has unique attributes. The embodiment of the present application does not limit the subcategory. Taking the animal category as an example, the subcategories of the animal category may be humans, cats, dogs, etc. Taking the plant category as an example, the subcategories of the plant category may be flowers, grass, trees, etc. Action predicates are used to characterize the actions or behaviors of the subject (such as "the children are playing").

[0138] For example, the fifth labels of the same category (such as animal category) are clustered to obtain the subcategories to which the fifth labels belong (such as people, cats, dogs), and the set of fifth labels belonging to the same subcategory (such as people) is determined as a cluster (such as the fifth labels corresponding to "you, me, him", where "you, me, him" belong to the same subcategory "people").

[0139] Through the embodiment of the present application, the fifth labels of the same category are clustered to remove labels with low confidence from the clusters, thereby improving the efficiency and accuracy of image processing.

[0140] In some embodiments, determining the third label from the cluster in step 2021B can be achieved by: weighting the image features included in the labels in the cluster to obtain a fourth image feature; weighting the recognition results included in the labels in the cluster to obtain a sixth recognition result; and constructing a third label based on the fourth image feature and the sixth recognition result.

[0141] For example, the image features included in the label A in the cluster are [0.14, 0.13, 0.06], and the image features included in the label B in the cluster are [0.52, 0.13, -0.06]. When each image feature has the same weight, that is, the weights of the image features included in the label A in the cluster and the label B in the cluster are both 0.5, the fourth image feature is [0.33, 0.13, 0]; the recognition result included in the label A in the cluster is 0.8, and the recognition result included in the label B in the cluster is 0.4. When each recognition result has the same weight, that is, the weights of the recognition results included in the label A in the cluster and the label B in the cluster are both 0.5, the sixth recognition result is 0.6.

[0142] In some embodiments, the above-mentioned "forming a third label based on the fourth image feature and the sixth recognition result" can be implemented in at least one of the following ways: determining the set of the fourth image feature and the sixth recognition result as the third label; or, determining the fourth image feature as the key of the first dictionary, and determining the sixth recognition result as the value of the first dictionary, obtaining the key-value pair of the first dictionary, and determining the first dictionary as the third label, wherein the key and value in the key-value pair are associated with each other.

[0143] For example, the set of the fourth image feature (such as [0.14, 0.13, 0.06]) and the sixth recognition result (such as [0.8, 0.1, 0.1]) is determined as the third label (such as [[0.14, 0.13, 0.06], [0.8, 0.1, 0.1]]); or, the fourth image feature is determined as the key of the first dictionary, and the sixth recognition result is determined as the value of the first dictionary to obtain the key-value pair of the first dictionary, and the first dictionary is determined as the third label (such as {[0.14, 0.13, 0.06], [0.8, 0.1, 0.1]}), wherein the key and value in the key-value pair are associated with each other.

[0144] In step 2022B, the fourth label in the cluster is deleted, and the third forward label library after the fourth label is deleted is used as the first forward label library.

[0145] Among them, the fourth label is different from the third label.

[0146] For example, the third label is label A in the cluster, and the fourth label (such as label B and label C) different from the third label in the cluster (taking cluster S as an example, cluster S includes [label A, label B, label C]) is deleted, and the third forward label library after deleting the fourth label is used as the first forward label library (such as cluster S in the first forward label library includes [label A]).

[0147] Through the embodiment of the present application, labels other than the third label in each cluster are deleted to reduce the waste of computing resources and further improve the efficiency of image processing.

[0148] In some embodiments, see Figure 3F , Figure 3F is a sixth flow chart of the image processing method provided in the embodiment of the present application, for Figure 3A Step 103 shown can be performed by Figure 3F Steps 1031 to 1035 are implemented as described in detail below.

[0149] In step 1031, the sixth tag in the first forward tag library is determined.

[0150] Here, the confidence of the recognition result included in the sixth tag is the maximum confidence among multiple confidences, and each confidence is the confidence of the recognition result included in each tag in the first forward tag library.

[0151] In some embodiments, step 1031 is consistent with the above step of “determining the second tag from the second forward tag library” and will not be described in detail here.

[0152] In step 1032, dot product processing is performed on the image feature included in the sixth tag and the first image feature to obtain a first dot product feature.

[0153] It should be noted that the dot product process is used to calculate the sum of the multiplication of corresponding elements of two vectors.

[0154] For example, given the image features included in the sixth label (such as [2, 3, 5]) and the first image features (such as [1, 4, 6]), dot product processing is performed on the image features included in the sixth label and the first image features to obtain a first dot product feature (such as 44).

[0155] In some embodiments, after step 1032, the following processing is performed: dynamically adapt the first dot product feature to obtain the dynamically adapted first dot product feature, and determine the product of the dynamically adapted first dot product feature and the first recognition result as the second recognition result.

[0156] Here, dynamic adaptation is used to determine the first dot product feature after dynamic adaptation in a non-parametric manner, which does not depend on a specific distribution form and does not require training through back propagation to determine the first dot product feature after dynamic adaptation, thereby saving computing resources and time.

[0157] In some embodiments, the above-mentioned “dynamically adapting the first dot product feature to obtain the first dot product feature after dynamic adaptation” can be implemented through steps 1033 to 1034.

[0158] In step 1033, a first difference between the preset feature and the first dot product feature is determined, and the product of the first difference and the first preset coefficient is determined as the first product.

[0159] For example, a first difference (eg, 0.2) between a preset feature (eg, 1) and a first dot product feature (eg, 0.8) is determined, and the product of the first difference and a first preset coefficient (eg, 5) is determined as a first product (eg, 1).

[0160] In step 1034, the product of the exponent of the first product and the second preset coefficient is determined as a second product.

[0161] In some embodiments, before step 1034, the following processing is performed: obtaining an exponential parameter, and performing an exponential operation on the first product based on the exponential parameter to obtain an exponent of the first product.

[0162] For example, based on an exponential parameter (such as 2), an exponent of the first product is determined (such as 2 to the power of 2 is 2).

[0163] Continuing with the above example, the product (eg, 0.4) of the exponent of the first product and the second preset coefficient (eg, 0.2) is determined as the first product (ie, the first dot product feature after dynamic adaptation).

[0164] It should be noted that the first preset coefficient is used to control the degree of emphasis on samples of different similarity degrees in the similarity calculation. The larger the value of the first preset coefficient, the higher the proportion of the first dot product feature in the second recognition result. The second preset coefficient is used to scale the numerical value of the first product to a uniform scale, that is, to adjust the intensity of dynamic adaptation, and scale the numerical value of the first dot product feature after dynamic adaptation to a uniform scale to avoid deviations caused by different features having different measurement units and numerical ranges.

[0165] In some embodiments, the above “determining the product of the first dot product feature after dynamic adaptation and the first recognition result as the second recognition result” can be implemented through step 1035 .

[0166] In step 1035, the product of the second product and the first recognition result is determined as the second recognition result.

[0167] For example, the product of the second product (eg, 0.4) and the first recognition result (eg, [0.8, 0.2]) is determined as the second recognition result (eg, [0.32, 0.09]).

[0168] Through the embodiments of the present application, the second forward label library is dynamically updated to reduce the waste of computing resources and improve the efficiency of the model. At the same time, labels with high uncertainty are removed to make it easy to determine the probability that the image belongs to the correct category, thereby improving the stability and accuracy of image recognition.

[0169] Continue to see Figure 3A In step 104, based on the negative label library and the first recognition result, a third recognition process is performed on the first image feature to obtain a third recognition result.

[0170] The confidence level of the recognition result corresponding to the third image feature included in the negative label library is less than the second confidence level threshold.

[0171] Here, the negative label library includes multiple labels, each label includes image features of historical images and recognition results of historical images, the confidence of the recognition results included in the labels is less than the second confidence threshold, the third recognition processing is a recognition processing method similar to the second recognition processing, and is used to identify objects, scenes or activities in the image, so as to easily determine the wrong category to which the first image belongs, and the third recognition result is used to characterize the probability that the first image belongs to different categories, wherein the probability that the first image belongs to the wrong category has a high proportion, and the third recognition result, the recognition result corresponding to the third image feature is based on the text feature of the image corresponding to the third image feature, and the first recognition processing is performed on the third image feature.

[0172] In some embodiments, see Figure 3G , Figure 3G is a seventh flow chart of the image processing method provided in the embodiment of the present application, Figure 3A Before step 104 shown, execute Figure 3G Steps 301 to 302 are described in detail below.

[0173] In step 301, the third image features of the third image and the text features of the third image are recognized to obtain a seventh recognition result, the third image is an image that has been processed, and the confidence of the seventh recognition result is less than the second confidence threshold.

[0174] Here, both the third image and the second image are images that have been processed. The embodiment of the present application does not limit the third image. The confidence of the seventh recognition result of the third image is less than the second confidence threshold and greater than or equal to the third confidence threshold, and the second confidence threshold is greater than the third confidence threshold.

[0175] In some embodiments, the above-mentioned "determining the third image" can be achieved in the following way: determining the eighth recognition result of the historical image based on the image features of the historical image and the text features of the historical image; when the confidence of the eighth recognition result of the historical image is less than the second confidence threshold and greater than or equal to the third confidence threshold, determining the historical image as the third image; when the confidence of the eighth recognition result of the historical image is greater than the first confidence threshold, determining the historical image as the second image.

[0176] It should be noted that the historical image includes the second image and the third image.

[0177] Continuing from the above embodiment, the step of “recognizing and processing the third image feature of the third image and the text feature of the third image to obtain a seventh recognition result” is similar to step 102 and will not be repeated here.

[0178] In step 302, a negative label library is constructed based on the third image feature of the third image and the seventh recognition result.

[0179] In some embodiments, step 302 can be implemented by: determining an initial negative label library and a seventh label, the seventh label including a third image feature of a third image and a seventh recognition result; based on the seventh recognition result, updating the initial negative label library to obtain a negative label library.

[0180] Continuing from the above embodiment, the step of "determining an initial negative label library" is similar to step 201A, and the step of "updating the initial negative label library based on the seventh recognition result to obtain a negative label library" is similar to step 202A, which will not be repeated here.

[0181] In some embodiments, see Figure 3H , Figure 3H is an eighth flow chart of the image processing method provided in the embodiment of the present application, for Figure 3A Step 104 shown can be performed by Figure 3H Steps 1041 to 1045 are implemented as described in detail below.

[0182] In step 1041 , the seventh label in the negative label library is determined.

[0183] Here, the confidence of the recognition result included in the seventh label is the minimum confidence among multiple confidences, and each confidence is the confidence of the recognition result included in each label in the negative label library.

[0184] In some embodiments, step 1041 is consistent with the above-mentioned step of “determining the second tag from the second forward tag library” and will not be described in detail here.

[0185] In step 1042, a dot product process is performed on the image feature included in the seventh tag and the first image feature to obtain a second dot product feature.

[0186] In some embodiments, step 1042 is similar to step 1032 and will not be described again herein.

[0187] In step 1043, a second difference between the preset feature and the second dot product feature is determined, and the product of the second difference and the first preset coefficient is determined as a third product.

[0188] In some embodiments, step 1043 is similar to step 1033 and will not be repeated here.

[0189] In step 1044, the product of the exponent of the third product and the second preset coefficient is determined as a fourth product.

[0190] In some embodiments, step 1044 is similar to step 1034 and will not be described again herein.

[0191] In step 1045, the product of the fourth product and the first recognition result is determined as the third recognition result.

[0192] In some embodiments, step 1045 is similar to step 1035 and is not described again herein.

[0193] Through the embodiments of the present application, the negative label library is dynamically updated to reduce the waste of computing resources and improve the efficiency of the model. At the same time, labels with high uncertainty are removed to make it easy to determine the probability that an image belongs to the correct category, thereby improving the stability and accuracy of image recognition.

[0194] Continue to see Figure 3A In step 105, the first recognition result, the second recognition result and the third recognition result are integrated to obtain a fourth recognition result of the first image.

[0195] In some embodiments, step 105 can be implemented by: determining the difference between the preset recognition result and the third recognition result as the eighth recognition result; weighting the first recognition result, the second recognition result and the eighth recognition result to obtain the fourth recognition result of the first image.

[0196] For example, the first recognition result is 0.4, the second recognition result is 0.8, and the third recognition result is 0.4. The difference between the preset recognition result (such as 1) and the third recognition result (such as 0.6) is determined as the eighth recognition result. When the first recognition result, the second recognition result and the eighth recognition result have the same weight, that is, the weights of the first recognition result, the second recognition result and the eighth recognition result are all 0.33, the fourth recognition result is 0.6. When the first recognition result, the second recognition result and the eighth recognition result have different weights, where the weight of the first recognition result is 0.6, the weight of the second recognition result is 0.2, and the weight of the eighth recognition result is 0.2, the fourth recognition result is 0.52.

[0197] Through the embodiments of the present application, the recognition results determined in different ways are combined to improve the generalization of image processing. At the same time, the adverse effects of noise on the recognition results are reduced, thereby improving the accuracy and reliability of image processing.

[0198] The following is an explanation of an exemplary application of the information processing method provided in an embodiment of the present application in a practical application scenario.

[0199] In the related art, by statically caching images and texts, image classification is not flexible enough when new data or distribution changes appear, which reduces the robustness of image classification.

[0200] In order to solve the above problems, the embodiment of the present application proposes an information processing method that can dynamically cache the image features and probability distribution of the image, better adapt to data changes, and further realize data denoising through the negative label library 4052, thereby improving the accuracy and robustness of information processing.

[0201] Taking image recognition as an example, see Figure 4 , Figure 4 This is a schematic diagram of the image recognition process provided in the embodiment of the present application. The image recognition process provided in the embodiment of the present application is explained below.

[0202] First, obtain a test image 401 (i.e., a first image) and a test label 402. Test image 401 is an input image during testing, and test label 402 (i.e., a label text of the first image) is an annotation text of the input image during testing. The annotation text is used to characterize the category information of the input image. Test label 402 can be implemented in the following way: replace the replacement text in the prompt word template with the category (such as cat) to obtain test label 402.

[0203] Secondly, through the image encoder 403 (such as E I ) Encode the test image and obtain the image representation vector (such as F I , i.e., the first image feature), and passes through the text encoder 404 (such as E T ) encodes the test label to obtain a text representation vector (such as F T , i.e. text features), the image encoder 403 and the text encoder 404 are both from the Contrastive Language-Image Pre-Training (CLIP) model based on contrastive text-image pairs.

[0204] Then, based on the image representation vector and the text representation vector, the classification result of the test image is calculated. The process of calculating the classification result of the test image is shown in formula (1).

[0205] P CLIP =sof tmax(H dot (F I , F T )) (1)

[0206] Among them, Hdot () is the dot product process, used to calculate F I and F T The dot product of the vectors is to calculate the similarity between the vectors. Softmax() is an activation function that converts a vector of multiple values ​​into a probability distribution. It is used to normalize the vectors of multiple values. Each value in the normalized probability distribution represents a category, and the sum of all values ​​is 1. By performing softmax() calculation on the result of the dot product, the similarity vector will become a probability table with a sum of 1), and the normalized similarity representation value is obtained. Finally, the category with the largest probability value is taken as the classification result of the image, P CLIP It is the first probability result (ie, the first recognition result), which is used to represent the probability distribution of the Contrastive Language-Image Pre-Training (CLIP) model in the classification task.

[0207] Finally, the final prediction result (i.e., the fourth recognition result) is determined based on the dynamic tag library 405, and the positive probability result P is calculated based on the dynamic tag library. positive (i.e. the second recognition result) and the negative probability result P negative (i.e., the third recognition result), and based on P CLIP Calculate the final prediction result P adapter , the calculation process is shown in formula (2).

[0208] P adapter =sum(P CLIP , P positive , P negative ) (2)

[0209] Among them, sum() is the addition process, P positive It is based on the forward tag library 4051 in the dynamic tag library and the image representation vector F I and the probability distribution P CLIP The calculated result is that the label of each category in the positive label library 4051 is the vector representation and probability distribution of an image, P positive It is used to represent the probability distribution that is easy to predict the correct category, P negative It is based on the negative label library 4052 in the dynamic label library and the image representation vector F I and the probability distribution P CLIP Calculated to represent a probability distribution where it is difficult to predict the correct class.

[0210] The dynamic label library 405 includes a positive label library 4051 (i.e., the first positive label library). Each category C in the positive label library 4051 can store at most M labels, and each label (i.e., dict) is represented by a dictionary, i.e., dict: {key, value}, where key is the key of the dictionary (i.e., the image representation vector F I ), value is the value of the dictionary (i.e., probability distribution P CLIP ), the update strategy of the forward tag library 4051 is described in detail below.

[0211] When the forward label library 4051 is initialized, the forward label library 4051 is empty (the empty forward label library 4051 is the second forward label library). When the number of labels of each category is less than M, the key directly uses the image representation vector F of the image. I (i.e. image features), value comes from the probability distribution P CLIP (i.e. recognition result), which is a onehot vector, called a positive pseudo label, denoted as l′ p When the number of labels under a certain category C (first category) in the label library reaches M (i.e., the first quantity threshold), if there is a new data of the same category, it is necessary to determine whether to update the label library based on the entropy value of the data of the same category. The judgment condition is shown in formula (3-5).

[0212] H dot (F I , F T )<H dot (Q high , F T ) (3)

[0213]

[0214] Among them, H dot () represents the self-entropy function, which is used to calculate the entropy value of a vector, Q high represents the image representation vector corresponding to the highest entropy value in the library, s and v represent the image representation vector F I And the image representation vector Q corresponding to the highest entropy value high , by comparing whether the entropy value of the new data is less than the highest entropy value in the forward label library 4051. If it is less than the highest entropy value in the forward label library 4051 (that is, greater than the first confidence level), the new data replaces the highest data (that is, the second label), thereby updating the data in the forward label library 4051.

[0215] At the same time, when the data in the forward label library 4051 is updated N times or the number of label data in the forward label library 4051 reaches a threshold, the K-means clustering method is used to update the labels in the forward label library 4051. The updating method is described in detail below.

[0216] Cluster all vector representations under a certain category C (i.e., the second category), perform automatic clustering according to the distance threshold (thresold), perform weighted average on the vector representations under the same cluster, obtain S new vector representations (i.e., the fourth image feature), perform weighted average on the dictionary values ​​corresponding to the vectors clustered into the same category, and obtain a new probability distribution (i.e., the sixth recognition result), where S is the number of clusters obtained by clustering, and calculate P positive As shown in formula (6).

[0217]

[0218] Among them, l′ p is a probability distribution, A(.) is a dynamic adaptation function, A(z) = α*exp(-β(1-z)), α and β are two hyperparameters, α (i.e., the second preset parameter) is used to control the intensity of the change. β (i.e., the first preset parameter) is used to adjust the sharpness of the similarity, that is, to control the importance of samples with different similarities in the similarity calculation. When β is large, the similarity distribution is more concentrated on a small number of highly similar samples. When β is small, the similarity distribution is relatively flat, and the output is a P with the probability distribution. CLIP The probability distribution of the same dimension strengthens the probability of predicting the correct category and reduces the probability of predicting other categories. 1 (i.e., the preset feature) is the preset parameter, and z is the feature input to the dynamic adaptation function. is the image representation vector F I And the image representation vector corresponding to the highest entropy value in the library The dot product processing (i.e. the first dot product feature) is performed.

[0219] The dynamic label library 405 includes a negative label library 4052. The update strategy of the negative label library 4052 is described in detail below.

[0220] In the process of P negative When calculating and updating the negative label library 4052, P negative The calculation method is similar to P positive is calculated in the same way, except that the probability distribution P negative is related to the probability distribution P positive Opposite probability distribution.

[0221] The updating method of the negative label library 4052 is basically the same as that of the positive label library 4051. The difference is that when updating the data in the negative label library 4052, a threshold is used to determine whether to add new data to the library. The judgment condition is shown in formula (7).

[0222]

[0223] in, is the first threshold of entropy value, It is the second threshold of the entropy value. The second threshold is greater than the first threshold, that is, when the entropy value is within the given threshold range, it needs to be added to the negative label library 4052 (negative label library).

[0224] It should be noted that the update method of the positive label library 4051 and the negative label library 4052 is to update according to the entropy value. When the entropy value of a new image is less than the maximum entropy value of the same category image in the positive label library 4051, the new image is added to the positive label library 4051 as a new image under the category. When the entropy value of the new image processes a certain threshold interval, the new image is added to the negative label library 4052 as a new image under the category. positive and P negative The calculation method is the same as that of negative After P negative Take the inversion so that P negative =1-P negative Finally, the total prediction result is P adapter =P CLIP +P positive +P negative .

[0225] In summary, the embodiment of the present application constructs a unique dynamic label generation architecture, collects and processes samples with certainty through the positive label library 4051 to generate positive pseudo labels, collects and processes samples with uncertainty through the negative label library 4052 to generate negative pseudo labels, and dynamically updates the samples to improve the efficiency and accuracy of the model, and through the positive pseudo labels, negative pseudo labels, and the data processing methods corresponding to the positive pseudo labels and negative pseudo labels, combined with the positive pseudo labels, negative pseudo labels and the results predicted by the pre-trained model, the probability distribution of the final classification is obtained, the adverse effects of noise on the estimated labels are reduced, the accuracy and reliability of pseudo labels are improved, the positive and negative labels are combined, the tolerance to noisy pseudo labels is enhanced, and the test data is better generalized to improve the performance and adaptability of the model. At the same time, a dynamic adaptation mechanism that does not require training is adopted, which can efficiently and effectively improve the performance and adaptability of the model during testing. Different uncertainty measurement indicators are used for sample screening and cache management.

[0226] The following further describes an exemplary structure of the image processing device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2 As shown, the software modules stored in the image processing device 555 of the memory 550 may include:

[0227] The preprocessing module 5551 is used to encode the first image to obtain the first image feature.

[0228] The recognition processing module 5552 is used to perform a first recognition processing on the first image feature to obtain a first recognition result; based on the first positive label library and the first recognition result, perform a second recognition processing on the first image feature to obtain a second recognition result, wherein the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than the first confidence threshold; based on the negative label library and the first recognition result, perform a third recognition processing on the first image feature to obtain a third recognition result, wherein the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than the second confidence threshold.

[0229] The result fusion module 5553 is used to fuse the first recognition result, the second recognition result and the third recognition result to obtain a fourth recognition result of the first image.

[0230] In some embodiments, the recognition processing module 5552 is also used to encode the second image to obtain a second image feature, where the second image is an image that has been processed; perform a first recognition process on the second image feature to obtain a fifth recognition result; when the confidence of the fifth recognition result is greater than the first confidence threshold, combine the second image feature and the fifth recognition result to obtain a first label for the second image; perform a first update process on the second forward label library based on the first label to obtain a first forward label library.

[0231] In some embodiments, the recognition processing module 5552 is also used to determine the first category of the first label; when the number of labels belonging to the first category in the second forward label library is less than a first quantity threshold, the first label is added to the second forward label library, and the second forward label library after adding the first label is used as the first forward label library; when the number of labels belonging to the first category in the second forward label library is greater than or equal to the first quantity threshold, the second forward label library is updated based on the confidence of the fifth recognition result, and the second forward label library after the first update processing is used as the first forward label library.

[0232] In some embodiments, the recognition processing module 5552 is also used to determine the second label from the second forward label library and determine the first confidence of the recognition result included in the second label; when the confidence of the fifth recognition result is greater than the first confidence, the second label in the second forward label library is replaced with the first label, and the second forward label library after replacing the second label is used as the first forward label library.

[0233] In some embodiments, the identification processing module 5552 is also used to determine a second category from the category to which the label in the third forward label library belongs when the first condition is met, and based on the second category, perform a second update process on the third forward label library to obtain the first forward label library, wherein the first condition includes one of the following: the number of labels in the third forward label library is greater than the second quantity threshold; the number of updates of the labels in the third forward label library is greater than or equal to the third quantity threshold.

[0234] In some embodiments, the identification processing module 5552 is also used to determine, for each cluster of the third forward label library, a third label from the cluster, where the cluster includes labels belonging to the second category; delete the fourth label in the cluster, and use the third forward label library after deleting the fourth label as the first forward label library, wherein the fourth label is different from the third label.

[0235] In some embodiments, the identification processing module 5552 is further used to perform clustering processing on each fifth tag belonging to the second category in the third forward tag library to obtain the subcategory to which each fifth tag belongs; and the fifth tags belonging to the same subcategory form a cluster.

[0236] In some embodiments, the recognition processing module 5552 is also used to perform weighted processing on the image features included in the labels in the cluster to obtain a fourth image feature; perform weighted processing on the recognition results included in the labels in the cluster to obtain a sixth recognition result; and construct a third label based on the fourth image feature and the sixth recognition result.

[0237] In some embodiments, the recognition processing module 5552 is also used to determine the sixth label in the first forward label library; perform dot product processing on the image feature included in the sixth label and the first image feature to obtain a first dot product feature; determine the first difference between the preset feature and the first dot product feature, and determine the product of the first difference and the first preset coefficient as the first product; determine the product of the exponent of the first product and the second preset coefficient as the second product; and determine the product of the second product and the first recognition result as the second recognition result.

[0238] In some embodiments, the recognition processing module 5552 is also used to determine the seventh label in the negative label library; perform dot product processing on the image feature included in the seventh label and the first image feature to obtain a second dot product feature; determine the second difference between the preset feature and the second dot product feature, and determine the product of the second difference and the first preset coefficient as the third product; determine the product of the exponent of the third product and the second preset coefficient as the fourth product; and determine the product of the fourth product and the first recognition result as the third recognition result.

[0239] In some embodiments, the recognition processing module 5552 is also used to perform recognition processing on the third image features of the third image and the text features of the third image to obtain a seventh recognition result, where the third image is an image that has been processed and the confidence of the seventh recognition result is less than the second confidence threshold; based on the third image features of the third image and the seventh recognition result, a negative label library is constructed.

[0240] An embodiment of the present application provides a computer program product, which includes computer executable instructions. The computer executable instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the image processing method described above in the embodiment of the present application.

[0241] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the image processing method provided by the present application embodiment, for example, 3A to 3H The image processing method is shown.

[0242] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0243] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0244] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0245] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0246] To summarize, based on the first positive label library and the first recognition result, the first image feature is subjected to a second recognition process to obtain a second recognition result, and based on the negative label library and the first recognition result, the first image feature is subjected to a third recognition process to obtain a third recognition result. Since the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than the first confidence threshold, and the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than the second confidence threshold, the first positive label library includes image features that are easy to obtain correct recognition results, and the negative label library includes image features that are easy to obtain incorrect recognition results. By using the image features included in the first positive label library and the image features included in the negative label library, the first image feature is recognized and processed, and the correct image features and the incorrect image features can be combined to assist the recognition process of the first image feature, thereby improving the generalization of image recognition. At the same time, the recognition results from different sources are integrated to improve the accuracy of image processing.

[0247] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Encoding the first image to obtain a first image feature; Performing a first recognition process on the first image feature to obtain a first recognition result; Based on the first positive label library and the first recognition result, performing a second recognition process on the first image feature to obtain a second recognition result, wherein the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than a first confidence threshold; Based on the negative label library and the first recognition result, performing a third recognition process on the first image feature to obtain a third recognition result, wherein the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than a second confidence threshold; The first recognition result, the second recognition result, and the third recognition result are integrated to obtain a fourth recognition result of the first image.

2. The method according to claim 1, characterized in that Before performing a second recognition process on the first image feature based on the first positive label library and the first recognition result to obtain the second recognition result, the method further includes: Encoding the second image to obtain a second image feature, where the second image is an image that has been processed; performing a first recognition process on the second image feature to obtain a fifth recognition result; When the confidence of the fifth recognition result is greater than the first confidence threshold, combining the second image feature and the fifth recognition result to obtain a first label for the second image; A first update process is performed on the second forward label library based on the first label to obtain the first forward label library.

3. The method according to claim 2, characterized in that The performing a first update process on the second forward label library based on the first label to obtain the first forward label library includes: determining a first category of the first tag; When the number of labels belonging to the first category in the second forward label library is less than a first quantity threshold, the first label is added to the second forward label library, and the second forward label library after the first label is added is used as the first forward label library; When the number of labels belonging to the first category in the second forward label library is greater than or equal to the first quantity threshold, a first update process is performed on the second forward label library based on the confidence of the fifth recognition result, and the second forward label library after the first update process is used as the first forward label library.

4. The method according to claim 3, characterized in that The step of performing a first update process on the second forward label library based on the confidence of the fifth recognition result, and using the second forward label library after the first update process as the first forward label library, comprises: Determine a second label from the second forward label library, and determine a first confidence level of a recognition result included in the second label; When the confidence of the fifth recognition result is greater than the first confidence, the second label in the second forward label library is replaced with the first label, and the second forward label library after the second label is replaced is used as the first forward label library.

5. The method according to claim 1, characterized in that Before performing a second recognition process on the first image feature based on the first positive label library and the first recognition result to obtain the second recognition result, the method further includes: When the first condition is met, determining a second category from the categories to which the labels in the third forward label library belong; Based on the second category, a second update process is performed on the third forward label library to obtain the first forward label library, wherein the first condition includes one of the following: the number of labels in the third forward label library is greater than a second number threshold; the number of updates of the labels in the third forward label library is greater than or equal to a third number threshold.

6. The method according to claim 5, characterized in that The step of performing a second update process on the third forward label library based on the second category to obtain the first forward label library includes: For each cluster of the third forward label library, determining a third label from the cluster, wherein the cluster includes labels belonging to the second category; The fourth label in the cluster is deleted, and the third forward label library after the fourth label is deleted is used as the first forward label library, wherein the fourth label is different from the third label.

7. The method according to claim 6, characterized in that The determining a third label from the cluster comprises: Performing weighted processing on the image features included in the labels in the cluster to obtain a fourth image feature; Performing weighted processing on the recognition results included in the tags in the cluster to obtain a sixth recognition result; The third label is constructed based on the fourth image feature and the sixth recognition result.

8. The method according to claim 1, characterized in that The performing a third recognition process on the first image feature based on the negative label library and the first recognition result to obtain a third recognition result includes: Determine a seventh label in the negative label library; Performing a dot product process on the image feature included in the seventh label and the first image feature to obtain a second dot product feature; Determine a second difference between the preset feature and the second dot product feature, and determine the product of the second difference and the first preset coefficient as a third product; Determine the product of the exponent of the third product and the second preset coefficient as a fourth product; The product of the fourth product and the first recognition result is determined as the third recognition result.

9. The method according to claim 1, characterized in that: Before performing a third recognition process on the first image feature based on the negative label library and the first recognition result to obtain the third recognition result, the method further includes: Performing recognition processing on a third image feature of a third image and a text feature of the third image to obtain a seventh recognition result, wherein the third image is an image that has been image processed, and a confidence level of the seventh recognition result is less than the second confidence level threshold; The negative label library is constructed based on the third image feature of the third image and the seventh recognition result.

10. An image processing device, characterized in that: The device comprises: A preprocessing module, used for encoding the first image to obtain a first image feature; a recognition processing module, configured to perform a first recognition processing on the first image feature to obtain a first recognition result; based on a first positive label library and the first recognition result, perform a second recognition processing on the first image feature to obtain a second recognition result, wherein the confidence of the recognition result corresponding to the second image feature included in the first positive label library is greater than a first confidence threshold; based on a negative label library and the first recognition result, perform a third recognition processing on the first image feature to obtain a third recognition result, wherein the confidence of the recognition result corresponding to the third image feature included in the negative label library is less than a second confidence threshold; The result fusion module is used to fuse the first recognition result, the second recognition result and the third recognition result to obtain a fourth recognition result of the first image.

11. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; The processor is used to implement the image processing method according to any one of claims 1 to 9 when executing the computer executable instructions or computer programs stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images