Image processing method, device and electronic equipment based on artificial intelligence

By extracting non-semantic and semantic features from images, establishing an index, and performing similarity query processing, the problem of low accuracy in similar image queries in existing technologies is solved, achieving image queries with higher accuracy and more efficient utilization of computational resources.

CN113821656BActive Publication Date: 2024-11-12TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202110640576.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-09
Publication Date
2024-11-12
Estimated Expiration
2041-06-09

AI Technical Summary

Technical Problem

In existing technologies, the accuracy of similar image queries is low, and it is impossible to effectively recall images with similar foregrounds but dissimilar backgrounds, and it is also impossible to prioritize the recall results.

Method used

By extracting non-semantic and semantic features from sample images, non-semantic and semantic indexes are established. Combined with non-semantic and semantic query processing, the accuracy of image queries and the utilization of computing resources are improved.

Benefits of technology

It improves the accuracy of similar image queries, can recall images with similar foregrounds but dissimilar backgrounds, and effectively sorts the recall results, thereby improving the utilization of computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113821656B_ABST
    Figure CN113821656B_ABST
Patent Text Reader

Abstract

The present application provides an image processing method, device, electronic device and computer-readable storage medium based on artificial intelligence; the method includes: determining the non-semantic features and semantic features of a sample image; establishing a non-semantic index according to the non-semantic features corresponding to a plurality of sample images, and establishing a semantic index according to the semantic features corresponding to a plurality of sample images; determining the non-semantic features and semantic features of a to-be-queried image; performing similarity query processing on the non-semantic index according to the non-semantic features of the to-be-queried image to obtain a similar non-semantic index, and performing similarity query processing on the semantic index according to the semantic features of the to-be-queried image to obtain a similar semantic index; using the sample images corresponding to the similar non-semantic index and at least part of the sample images corresponding to the similar semantic index as similar sample images of the to-be-queried image. Through the present application, the accuracy of similar image query can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to computer vision technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium based on artificial intelligence. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Computer Vision (CV) is an important branch of artificial intelligence, which mainly studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data.

[0003] Similar image query is an important application of computer vision technology. In the solutions provided by related technologies, the model is usually trained by similarity metric learning, the image features of the image are predicted based on the trained model, and similar image queries are performed based on the image features. However, since the similarity metric learning method indiscriminately focuses on all features in the image, it is impossible to query images with similar foregrounds and dissimilar backgrounds when performing similar image queries, that is, the query accuracy is low. Summary of the invention

[0004] The embodiments of the present application provide an artificial intelligence-based image processing method, device, electronic device, and computer-readable storage medium, which can improve the accuracy of similar image queries.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides an image processing method based on artificial intelligence, including:

[0007] Performing feature extraction processing on the sample image to obtain non-semantic features and semantic features of the sample image;

[0008] Establishing a non-semantic index according to the non-semantic features respectively corresponding to the plurality of sample images, and establishing a semantic index according to the semantic features respectively corresponding to the plurality of sample images;

[0009] Performing feature extraction processing on the query image to obtain non-semantic features and semantic features of the query image;

[0010] Performing similarity query processing on the non-semantic index according to the non-semantic features of the image to be queried to obtain a similar non-semantic index, and performing similarity query processing on the semantic index according to the semantic features of the image to be queried to obtain a similar semantic index;

[0011] The sample images corresponding to the similar non-semantic indexes and at least part of the sample images corresponding to the similar semantic indexes are used as similar sample images of the image to be queried.

[0012] The present application provides an artificial intelligence-based image processing device, comprising:

[0013] A feature extraction module is used to perform feature extraction processing on the sample image to obtain non-semantic features and semantic features of the sample image;

[0014] An index building module, used to build a non-semantic index according to the non-semantic features respectively corresponding to the plurality of sample images, and to build a semantic index according to the semantic features respectively corresponding to the plurality of sample images;

[0015] The feature extraction module is further used to perform feature extraction processing on the query image to obtain non-semantic features and semantic features of the query image;

[0016] A similarity query module, used for performing similarity query processing on the non-semantic index according to the non-semantic features of the image to be queried to obtain a similar non-semantic index, and performing similarity query processing on the semantic index according to the semantic features of the image to be queried to obtain a similar semantic index;

[0017] The similarity query module is further configured to use the sample images corresponding to the similar non-semantic indexes and at least part of the sample images corresponding to the similar semantic indexes as similar sample images of the image to be queried.

[0018] An embodiment of the present application provides an electronic device, including:

[0019] A memory for storing executable instructions;

[0020] The processor is used to implement the artificial intelligence-based image processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the artificial intelligence-based image processing method provided in the embodiment of the present application.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] Feature extraction is performed on sample images to obtain non-semantic features and semantic features, and a non-semantic index is established based on the non-semantic features corresponding to multiple sample images. At the same time, a semantic index is established based on the semantic features corresponding to multiple sample images. In this way, similar image queries are implemented for the image to be queried in both non-semantic and semantic branches, which can simultaneously focus on features of the image in different aspects, thereby improving query accuracy, and also improving the actual utilization rate of computing resources consumed by electronic devices in the image processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of the architecture of an image processing system based on artificial intelligence provided in an embodiment of the present application;

[0025] Figure 2 It is a schematic diagram of the architecture of a terminal device provided in an embodiment of the present application;

[0026] Figure 3A It is a flowchart of an image processing method based on artificial intelligence provided in an embodiment of the present application;

[0027] Figure 3B It is a flowchart of an image processing method based on artificial intelligence provided in an embodiment of the present application;

[0028] Figure 3C It is a flowchart of an image processing method based on artificial intelligence provided in an embodiment of the present application;

[0029] Figure 3D It is a flowchart of an image processing method based on artificial intelligence provided in an embodiment of the present application;

[0030] Figure 3E It is a flowchart of an image processing method based on artificial intelligence provided in an embodiment of the present application;

[0031] Figure 4 is a schematic diagram of a bucket query provided in an embodiment of the present application;

[0032] Figure 5 is a schematic diagram of an image processing method based on artificial intelligence provided in an embodiment of the present application;

[0033] Figure 6 It is a schematic diagram of the architecture of the image processing model provided in the embodiment of the present application;

[0034] Figure 7 It is a schematic diagram of the storyboard fragment provided in the embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second\third" are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In the following description, the term "plurality" refers to at least two.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) Image: The embodiments of the present application mainly involve three types of images, namely, sample images, images to be queried, and training images. Sample images refer to samples used to support similar image queries, for example, stored images can be used as sample images, or images in stored videos can be used as sample images; images to be queried are used to trigger similar image queries, that is, the purpose of similar image queries is to find similar sample images of the images to be queried among multiple sample images; training images are used to train image processing models, wherein the training images can be the same as or different from the sample images.

[0041] 2) Semantics: represents the actual meaning of the image, such as the semantic category to which the content in the image belongs, such as people, cats, dogs, etc.

[0042] 3) Metric Learning: Given some similar images and some dissimilar images, by learning the similarity (distance) metric, the similarity between originally similar images increases (or the distance decreases), while the similarity between originally dissimilar images decreases (or the distance increases).

[0043] 4) Image processing model: A model constructed based on artificial intelligence principles for feature extraction processing, wherein the type of image processing model is not limited, for example, it can be an artificial neural network model. In an embodiment of the present application, the image processing model may include an initial feature extraction network, a non-semantic branch network, and a semantic branch network, wherein the initial feature extraction network is shared by the non-semantic branch network and the semantic branch network, the non-semantic branch network includes a non-semantic feature extraction network, and the semantic branch network includes a semantic feature extraction network and a semantic classification network. It is worth noting that the network in the image processing model refers to an artificial neural network, not a communication network.

[0044] 5) Loss value: used to indicate the difference between the output of the model and the actual result (or labeled result). The training of the model mainly involves two processes: forward propagation and back propagation. Taking the neural network model including input layer, hidden layer and output layer as an example, forward propagation processing refers to processing through the input layer, hidden layer and output layer in sequence to finally obtain the output result; back propagation processing refers to propagating the calculated loss value to the output layer, hidden layer and input layer in sequence, thereby updating the weight parameters in each layer.

[0045] 6) Storyboard processing: also known as scene segmentation processing, refers to dividing the video into multiple storyboard segments, each of which includes multiple similar images.

[0046] 7) Intersection over Union (IoU): For any two sets, the IoU may refer to the ratio of the number of elements included in the intersection of the two sets to the number of elements included in the union of the two sets. In the embodiment of the present application, the IoU may be used as a sorting score to achieve sorting of multiple similar sample videos.

[0047] 8) Clustering: the process of dividing a set of physical or abstract objects into multiple clusters consisting of similar objects. The present application embodiment does not limit the method of clustering, for example, it can be a Kmeans algorithm, a hierarchical clustering algorithm, or a K-means++ algorithm.

[0048] For similar image queries, in the solutions provided by related technologies, a similarity (distance) metric learning method is usually used to train the model. The trained model predicts the image features of multiple sample images and the image features of the image to be queried. Then, based on the similarity (distance) between the image features of the image to be queried and the image features of the sample images, similar sample images that are similar to the image to be queried are recalled from the multiple sample images.

[0049] However, the similarity metric learning method indiscriminately focuses on all features in the image, resulting in the inability to recall sample images that are similar to the foreground of the query image but dissimilar to the background according to the solution provided by the related technology, resulting in low accuracy of similar image queries. For example, in some application scenarios, a malicious attacker replaces the background in image A by cutting out the image to obtain image B. If the solution provided by the related technology is used, image A and image B will be mistakenly identified as dissimilar.

[0050] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium based on artificial intelligence, which can improve the accuracy of similar image query and ensure that the computing resources consumed by the electronic device in the image processing process are effectively utilized. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminal devices or as a server.

[0051] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of an artificial intelligence-based image processing system 100 provided in an embodiment of the present application, wherein a terminal device 400 is connected to a server 200 via a network 300, and the server 200 is connected to a database 500, wherein the network 300 may be a wide area network or a local area network, or a combination of the two.

[0052] In some embodiments, taking the electronic device as a terminal device as an example, the artificial intelligence-based image processing method provided in the embodiments of the present application can be implemented by the terminal device. For example, the terminal device 400 performs feature extraction processing on the sample image to obtain the non-semantic features and semantic features of the sample image, establishes a non-semantic index based on the non-semantic features corresponding to the multiple sample images, and establishes a semantic index based on the semantic features corresponding to the multiple sample images. When the terminal device 400 receives a request for a similar image query, it performs feature extraction processing on the image to be queried corresponding to the request to obtain the non-semantic features and semantic features of the image to be queried. As an example, in Figure 1The option of "Similar Image Query" is shown in the figure. When a trigger operation for this option is received, it is determined that a request for similar image query is received. Then, the terminal device 400 performs similar query processing on the non-semantic index according to the non-semantic features of the image to be queried to obtain a similar non-semantic index, and performs similar query processing on the semantic index according to the semantic features of the image to be queried to obtain a similar semantic index. Finally, the terminal device 400 uses the sample images corresponding to the similar non-semantic indexes and at least part of the sample images corresponding to the similar semantic indexes as similar sample images of the image to be queried. As an example, in Figure 1 exemplarily shows a query image and a similar sample image, both of which include airplanes.

[0053] It is worth noting that the terminal device 400 can train the image processing model through the training image, and implement feature extraction processing of the sample image and the image to be queried according to the trained image processing model, wherein the trained image processing model can be stored locally in the terminal device 400 and called when needed. In addition, for each of the training images, sample images and images to be queried, the images can be pre-stored locally by the terminal device 400, can be taken in real time by the terminal device 400, or can be obtained by the terminal device 400 from the outside (such as the server 200, the database 500 or the blockchain, etc.).

[0054] In some embodiments, taking the electronic device as a server as an example, the artificial intelligence-based image processing method provided in the embodiment of the present application can also be implemented by the server. For example, the server 200 performs a series of processing on the sample image to obtain a non-semantic index and a semantic index, and stores the non-semantic index and the semantic index locally, such as in a database 500. When the server 200 receives a request for a similar image query, a series of processing is performed on the image to be queried corresponding to the request, and the recalled similar sample image is obtained by combining the non-semantic index and the semantic index.

[0055] Similarly, the server 200 can also train the image processing model through the training images, and implement feature extraction processing of the sample image and the image to be queried according to the trained image processing model, wherein the trained image processing model can be stored locally on the server 200, such as in the distributed file system of the server 200. The server 200 can also send the trained image processing model to the terminal device 400, so that the terminal device 400 has the capability of feature extraction processing. In addition, each of the training images, sample images and the image to be queried can be pre-stored locally by the server 200, or can be obtained from the outside by the server 200.

[0056] In some embodiments, the artificial intelligence-based image processing method provided in the embodiments of the present application can also be implemented collaboratively by a terminal device and a server. For example, the server 200 performs a series of processing on the sample image to obtain a non-semantic index and a semantic index, and stores the non-semantic index and the semantic index locally. When the terminal device 400 receives a request for a similar image query, it sends the image to be queried corresponding to the request to the server 200. The server 200 performs a series of processing on the received image to be queried, combines the non-semantic index and the semantic index to obtain a recalled similar sample image, and sends the similar sample image to the terminal device 400, so that the terminal device 400 responds to the request for a similar image query based on the received similar sample image.

[0057] In some embodiments, various results involved in the image processing process (such as training images, sample images, images to be queried, non-semantic indexes, semantic indexes, etc.) can be stored in the blockchain. Since the blockchain has the characteristic of being tamper-proof, the accuracy of the data in the blockchain can be guaranteed. The electronic device can send a query request to the blockchain to query the data stored in the blockchain.

[0058] In some embodiments, the terminal device 400 or the server 200 can implement the image processing method based on artificial intelligence provided by the embodiment of the present application by running a computer program. The computer program can be Figure 1 The client 410 is shown. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0059] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, wherein the cloud service may be an image processing service based on artificial intelligence for the terminal device 400 to call. The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a smart watch, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0060] In some embodiments, the database 500 and the server 200 can be independently configured. In some embodiments, the database 500 and the server 200 can also be integrated together, that is, the database 500 can be regarded as existing inside the server 200 and integrated with the server 200, and the server 200 can provide the data management function of the database 500.

[0061] Taking the electronic device provided in the embodiment of the present application as a terminal device as an example, it can be understood that in the case where the electronic device is a server, Figure 2 Some of the structures shown in FIG. 1 (such as the user interface, the presentation module, and the input processing module) may be omitted. Figure 2 , Figure 2 is a schematic diagram of the structure of a terminal device 400 provided in an embodiment of the present application, Figure 2 The terminal device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0062] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0063] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0064] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0065] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0066] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0067] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0068] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0069] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0070] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0071] In some embodiments, the artificial intelligence-based image processing device provided in the embodiments of the present application can be implemented in software. Figure 2 An artificial intelligence-based image processing device 455 stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, including the following software modules: a feature extraction module 4551, an index building module 4552, and a similarity query module 4553. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0072] The artificial intelligence-based image processing method provided in the embodiments of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided in the embodiments of the present application.

[0073] See also Figure 3A , Figure 3A is a flowchart of an image processing method based on artificial intelligence provided by an embodiment of the present application. The method can be executed by an electronic device. Figure 3A The steps shown are explained.

[0074] In step 101, feature extraction processing is performed on a sample image to obtain non-semantic features and semantic features of the sample image.

[0075] Here, a plurality of sample images are obtained, and feature extraction processing is performed on each of the sample images. The feature extraction processing includes two aspects: the first aspect is to extract non-semantic features of the sample images, and the second aspect is to extract semantic features of the sample images.

[0076] The embodiments of the present application do not limit the form of non-semantic features and semantic features, for example, both can be in vector form; the implementation method of feature extraction processing is also not limited, for example, it can be implemented through a specific image processing model.

[0077] In step 102, a non-semantic index is established according to the non-semantic features respectively corresponding to the plurality of sample images, and a semantic index is established according to the semantic features respectively corresponding to the plurality of sample images.

[0078] Here, the non-semantic index may be a non-semantic feature corresponding to at least part of the sample image, and the semantic index may be a semantic feature corresponding to at least part of the sample image.

[0079] In some embodiments, the above-mentioned establishment of a non-semantic index based on the non-semantic features corresponding to multiple sample images and the establishment of a semantic index based on the semantic features corresponding to multiple sample images can be achieved in the following manner: the non-semantic features corresponding to all sample images are used as non-semantic indexes, and the semantic features corresponding to all sample images are used as semantic indexes. In this case, there is a one-to-one correspondence between the sample images and the non-semantic indexes, and between the sample images and the semantic indexes.

[0080] In some embodiments, the above-mentioned establishment of a non-semantic index based on the non-semantic features corresponding to multiple sample images and establishment of a semantic index based on the semantic features corresponding to multiple sample images can be achieved in the following manner: clustering the non-semantic features corresponding to multiple sample images, and using the obtained multiple non-semantic clustering centers as non-semantic indexes; clustering the semantic features corresponding to multiple sample images, and using the obtained multiple semantic clustering centers as semantic indexes.

[0081] Here, in order to avoid the excessive amount of subsequent calculations caused by too many indexes, the non-semantic features corresponding to all sample images can also be clustered, and the multiple cluster centers obtained (named as non-semantic cluster centers for easy distinction) are all used as non-semantic indexes; at the same time, the semantic features corresponding to all sample images are clustered, and the multiple cluster centers obtained (named as semantic cluster centers for easy distinction) are all used as semantic indexes. The embodiment of the present application does not limit the method of clustering processing. For example, clustering processing can be achieved by algorithms such as Kmeans algorithm, hierarchical clustering algorithm or K-means++ algorithm.

[0082] Through the above method, it can be ensured that the number of non-semantic indexes and the number of semantic indexes will not be too large, and the determined non-semantic indexes and semantic indexes are representative, which is helpful to improve the subsequent similar query effect.

[0083] In some embodiments, after step 102, it also includes: determining the non-semantic distances between the non-semantic features of the sample image and multiple non-semantic indexes respectively, and establishing a correspondence between the sample image and the non-semantic indexes that satisfy the non-semantic distance conditions; determining the semantic distances between the semantic features of the sample image and multiple semantic indexes respectively, and establishing a correspondence between the sample image and the semantic indexes that satisfy the semantic distance conditions.

[0084] In the embodiment of the present application, the correspondence between the sample image and the non-semantic index, and the correspondence between the sample image and the semantic index can also be intelligently established.

[0085] For example, for each sample image, the distance between the non-semantic features of the sample image and all non-semantic indexes (hereinafter referred to as non-semantic distance for easy distinction) can be determined respectively, and a corresponding relationship between the sample image and the non-semantic index that meets the non-semantic distance condition is established, wherein the non-semantic distance condition includes at least one of the following: the non-semantic distance is less than the non-semantic distance threshold; when the non-semantic distance is sorted in ascending order, the non-semantic distance is ranked in the top N. The non-semantic distance threshold and N can be set according to the actual application scenario, such as setting N to 1.

[0086] For each sample image, the distance between the semantic features of the sample image and all semantic indexes (hereinafter named as semantic distance for easy distinction) can be determined respectively, and a corresponding relationship between the sample image and the semantic index that meets the semantic distance condition is established, wherein the semantic distance condition includes at least one of the following: the semantic distance is less than the semantic distance threshold; when the semantic distances are sorted in ascending order, the semantic distance is ranked in the top N places.

[0087] It is worth noting that in the embodiment of the present application, the distance between features can be L2 distance or other types of distance. The smaller the distance between features, the more similar the two features are.

[0088] Through the above manner, it is possible to ensure that each sample image establishes a corresponding relationship with at least one non-semantic index and at the same time establishes a corresponding relationship with at least one semantic index, and the accuracy of the established corresponding relationship can be guaranteed.

[0089] In step 103, feature extraction processing is performed on the query image to obtain non-semantic features and semantic features of the query image.

[0090] Here, the manner of performing feature extraction processing on the query image may be the same as step 101. After the feature extraction processing, non-semantic features and semantic features of the query image are obtained.

[0091] It is worth noting that the embodiment of the present application does not limit the execution order between step 101 and step 103. For example, they can be executed simultaneously or one after the other.

[0092] In step 104, similarity query processing is performed on the non-semantic index according to the non-semantic features of the image to be queried to obtain a similar non-semantic index, and similarity query processing is performed on the semantic index according to the semantic features of the image to be queried to obtain a similar semantic index.

[0093] For example, based on the non-semantic distance between the non-semantic features of the image to be queried and each non-semantic index, at least some of the non-semantic indexes can be screened out as similar non-semantic indexes; based on the semantic distance between the semantic features of the image to be queried and each semantic index, at least some of the semantic indexes can be screened out as similar semantic indexes. The embodiment of the present application does not limit the number of similar non-semantic indexes determined, for example, it can be at least one, and the same applies to similar semantic indexes.

[0094] In some embodiments, the above-mentioned similarity query processing of non-semantic indexes according to the non-semantic features of the image to be queried to obtain a similar non-semantic index, and similarity query processing of semantic indexes according to the semantic features of the image to be queried to obtain a similar semantic index can be achieved in the following manner: respectively determine the non-semantic distances between the non-semantic features of the image to be queried and multiple non-semantic indexes, and use the non-semantic indexes that meet the non-semantic distance conditions as similar non-semantic indexes; respectively determine the semantic distances between the semantic features of the image to be queried and multiple semantic indexes, and use the semantic indexes that meet the semantic distance conditions as similar semantic indexes.

[0095] Here, the non-semantic distance between the non-semantic features of the image to be queried and all non-semantic indexes can be determined respectively, and the non-semantic indexes that meet the non-semantic distance condition can be used as similar non-semantic indexes; the semantic distance between the semantic features of the image to be queried and all semantic indexes can be determined respectively, and the semantic indexes that meet the semantic distance condition can be used as similar semantic indexes. Among them, the non-semantic distance condition and the semantic distance condition can be referred to the above content. In the above manner, fast and effective index screening can be performed based on distance.

[0096] In step 105, sample images corresponding to similar non-semantic indexes and at least part of sample images corresponding to similar semantic indexes are used as similar sample images of the image to be queried.

[0097] Here, the sample images corresponding to the similar non-semantic indexes and the sample images corresponding to the similar semantic indexes may be used as similar sample images (similar images) of the image to be queried.

[0098] In some embodiments, the sample images can also be further screened, for example, all sample images corresponding to similar non-semantic indexes are determined, and the non-semantic distance between the non-semantic features of these sample images and the non-semantic features of the image to be queried is determined, and the sample images that meet the non-semantic distance condition are used as similar sample images; all sample images corresponding to similar semantic indexes are determined, and the semantic distance between the semantic features of these sample images and the semantic features of the image to be queried is determined, and the sample images that meet the semantic distance condition are used as similar sample images. In this way, the query accuracy can be further improved.

[0099] like Figure 3A As shown, the embodiment of the present application combines non-semantic indexing and semantic indexing to implement similar image query, which can improve the query accuracy. For example, it can query sample images with a foreground similar to the image to be queried but a background different from that of the image to be queried; at the same time, it can also improve the actual utilization rate of the computing resources consumed by the electronic device in the image processing process.

[0100] In some embodiments, see Figure 3B , Figure 3B is a flowchart of an image processing method based on artificial intelligence provided by an embodiment of the present application. The method can be executed by an electronic device. Figure 3A Before step 101, in step 201, feature extraction processing can be performed on the training images in the training image group through the image processing model to obtain non-semantic features and semantic features of the training images, and semantic classification processing can be performed on the semantic features of the training images to obtain predicted classification features.

[0101] In an embodiment of the present application, feature extraction processing of sample images and images to be queried can be implemented through an image processing model. In order to ensure the accuracy of feature extraction processing, the image processing model can be pre-trained.

[0102] During the training process, for each training image in the training image group, the image processing model performs forward propagation processing, that is, the training image is subjected to feature extraction processing to obtain the non-semantic features and semantic features of the training image, and the semantic features of the training image are subjected to semantic classification processing to obtain the predicted classification features. The training image group includes multiple training images, and the training image group can be generated manually or automatically; the predicted classification features are used to represent the predicted semantic classification of the training image, for example, the predicted classification features can be a predicted probability vector, in which the semantic classification corresponding to the largest value is the predicted semantic classification.

[0103] In step 202, loss calculation processing is performed according to the non-semantic features, semantic features and predicted classification features of the training images, respectively, to obtain the non-semantic feature loss value, semantic feature loss value and semantic classification loss value of the corresponding training image group.

[0104] Here, corresponding loss functions can be set in advance for non-semantic features, semantic features, and predicted classification features, respectively. In this way, loss calculation processing can be performed based on the non-semantic features of the training image to obtain the non-semantic feature loss value of the training image group; loss calculation processing can be performed based on the semantic features of the training image to obtain the semantic feature loss value of the training image group; loss calculation processing can be performed based on the predicted classification features of the training image to obtain the semantic classification loss value of the training image group. The embodiment of the present application does not limit the type of loss function, for example, including but not limited to the cross entropy loss function, triplet loss function (Triplet Loss) and relative entropy (also known as Kullback-Leibler divergence, i.e. KL divergence) loss function, etc.

[0105] In some embodiments, the training image group includes a first training image, a second training image similar to the first training image, and a third training image dissimilar to the first training image; the above-mentioned loss calculation processing according to the non-semantic features, semantic features and predicted classification features of the training images can be implemented in the following manner to obtain the non-semantic feature loss value, semantic feature loss value and semantic classification loss value of the corresponding training image group: the non-semantic feature loss value of the training image group is determined according to the non-semantic distance between the non-semantic features of the first training image and the non-semantic features of the second training image, and the non-semantic distance between the non-semantic features of the first training image and the non-semantic features of the third training image; the semantic feature loss value of the training image group is determined according to the semantic distance between the semantic features of the first training image and the semantic features of the second training image, and the semantic distance between the semantic features of the first training image and the semantic features of the third training image; the semantic classification loss value of any training image in the training image group is determined according to the labeled classification features and predicted classification features of any training image, and the semantic classification loss values ​​corresponding to the first training image, the second training image and the third training image are fused to obtain the semantic classification loss value of the training image group.

[0106] In an embodiment of the present application, the training image group may be a triplet, i.e., including a first training image, a second training image similar to the first training image, and a third training image dissimilar to the first training image. In this case, the non-semantic feature loss value of the training image group may be determined based on the non-semantic distance between the non-semantic features of the first training image and the non-semantic features of the second training image, and the non-semantic distance between the non-semantic features of the first training image and the non-semantic features of the third training image. The non-semantic feature loss value is used to enlarge the difference (i.e., distance) between the first training image and the third training image in non-semantic features, while reducing the difference between the first training image and the second training image in non-semantic features, wherein the non-semantic feature loss value may be calculated by Triplet loss, but is not limited thereto.

[0107] Similarly, the semantic feature loss value of the training image group can be determined based on the semantic distance between the semantic features of the first training image and the semantic features of the second training image, and the semantic distance between the semantic features of the first training image and the semantic features of the third training image.

[0108] As for the predicted classification features, for each training image in the training image group, the semantic classification loss value of the training image can be determined based on the labeled classification features and predicted classification features of the training image, wherein the semantic classification loss value can be calculated by the relative entropy loss function, but is not limited to this. Then, the semantic classification loss values ​​corresponding to the first training image, the second training image, and the third training image in the training image group are fused to obtain the semantic classification loss value of the training image group. In the embodiment of the present application, the fusion process can be achieved by direct summation or weighted summation.

[0109] In the case where the training image group is a triplet, the non-semantic feature loss value, semantic feature loss value and semantic classification loss value of the triplet can be determined quickly and accurately through the above method.

[0110] In some embodiments, before determining the semantic classification loss value of any training image in the training image group based on the labeled classification features and predicted classification features of any training image, it also includes: performing any of the following processing: obtaining the labeled classification features set for any training image; performing semantic classification processing on any training image through a semantic classification model to obtain the labeled classification features; wherein the semantic classification model is different from the image processing model.

[0111] Here, for the training images in the training image group, the labeled classification features can be obtained in the following two ways.

[0112] The first method is to obtain the labeled classification features set for the training images, that is, the labeled classification features can be manually labeled. This method can ensure the absolute accuracy of the labeled classification features.

[0113] The second method is to perform semantic classification processing on the training images through a semantic classification model to obtain labeled classification features, where the semantic classification model is different from the image processing model. This method can ensure the accuracy of the labeled classification features to a certain extent, and compared with the first method, it can save the cost of manual labeling.

[0114] The embodiment of the present application does not limit the type of semantic classification model, for example, it can be a neural network model, etc. Before the semantic classification model is used to perform semantic classification processing on the training image, the semantic classification model can also be pre-trained, for example, by training with an open source image set, so that the accuracy of the obtained labeled classification features can be improved.

[0115] In step 203, the non-semantic feature loss values, semantic feature loss values, and semantic classification loss values ​​of the training image group are fused, and the image processing model is trained based on the obtained fusion loss values.

[0116] Here, the non-semantic feature loss value, semantic feature loss value and semantic classification loss value of the training image group are fused, and back-propagation processing is performed in the image processing model according to the obtained fusion loss value to realize the training of the image processing model.

[0117] It is worth noting that the training process of the image processing model is essentially the process of updating the weight parameters in the image processing model.

[0118] exist Figure 3B middle, Figure 3A The illustrated step 101 may be updated to step 204. In step 204, feature extraction processing is performed on the sample image by using the trained image processing model to obtain non-semantic features and semantic features of the sample image.

[0119] exist Figure 3B middle, Figure 3A The illustrated step 103 may be updated to step 205. In step 205, feature extraction processing is performed on the query image through the trained image processing model to obtain non-semantic features and semantic features of the query image.

[0120] When the image processing model is trained, the trained image processing model can be used to extract features from sample images and images to be queried.

[0121] like Figure 3B As shown, the embodiment of the present application trains the image processing model based on the principle of artificial intelligence, and performs feature extraction processing through the trained image processing model, which can effectively improve the accuracy of feature extraction processing.

[0122] In some embodiments, see Figure 3C , Figure 3C is a flowchart of an artificial intelligence-based image processing method provided in an embodiment of the present application, which can be executed by an electronic device. Figure 3B The illustrated step 201 can be implemented through steps 301 to 304 , which will be described in conjunction with each step.

[0123] In step 301, an initial feature extraction network is used to perform initial feature extraction processing on a training image to obtain initial features of the training image.

[0124] In an embodiment of the present application, the image processing model may mainly include three parts: an initial feature extraction network (also called a basic feature extraction network), a non-semantic branch network and a semantic branch network, wherein the non-semantic branch network and the semantic branch network share the initial feature extraction network; the non-semantic branch network includes a non-semantic feature extraction network; and the semantic branch network includes a semantic feature extraction network and a semantic classification network.

[0125] In the process of forward propagation processing of the training image through the image processing model, the initial feature extraction network is first used to perform initial feature extraction processing on the training image to obtain the initial features of the training image.

[0126] It is worth noting that, before step 301, the initial feature extraction network may be trained separately, such as by training with an open source image set, so as to improve the overall training effect of the image processing model.

[0127] In step 302, a non-semantic feature extraction network is used to perform non-semantic feature extraction processing on the initial features of the training image to obtain non-semantic features of the training image.

[0128] Here, the output result of the initial feature extraction network (ie, the initial feature) is subjected to non-semantic feature extraction processing by the non-semantic feature extraction network to obtain the non-semantic features of the training image.

[0129] In step 303, a semantic feature extraction network is used to perform semantic feature extraction processing on the initial features of the training image to obtain the semantic features of the training image.

[0130] Here, the output result of the initial feature extraction network is processed by semantic feature extraction network to obtain the semantic features of the training image.

[0131] In step 304, semantic classification processing is performed on the semantic features of the training image through a semantic classification network to obtain predicted classification features of the training image.

[0132] Here, the output result (ie, semantic feature) of the semantic feature extraction network is semantically classified by the semantic classification network to obtain the predicted classification features of the training image.

[0133] exist Figure 3C middle, Figure 3B The illustrated step 203 may be implemented through steps 305 and 306, or through steps 305 and 307, which will be described in conjunction with each step.

[0134] In step 305, the non-semantic feature loss value, the semantic feature loss value and the semantic classification loss value of the training image group are fused to obtain a fused loss value.

[0135] In step 306, the non-semantic feature extraction network, the semantic feature extraction network, and the semantic classification network are trained according to the fusion loss value.

[0136] The embodiment of the present application provides two training methods. The first training method is to use the non-semantic feature extraction network, the semantic feature extraction network and the semantic classification network as the networks that need to be trained, and train them according to the fusion loss value. This method has a high training efficiency.

[0137] In step 307, the initial feature extraction network, the non-semantic feature extraction network, the semantic feature extraction network and the semantic classification network are trained according to the fusion loss value.

[0138] The second training method is to use the initial feature extraction network, non-semantic feature extraction network, semantic feature extraction network and semantic classification network as the networks that need to be trained, and train them according to the fusion loss value. This method has better training effect, but it may take longer training time than the first training method.

[0139] In some embodiments, during the training of the image processing model, the training rate of the semantic classification network is higher than that of the network in the image processing model that is distinct from the semantic classification network.

[0140] In the training process of the image processing model, compared with other networks, the semantic classification network is more prone to overfitting (because the task of semantic classification is to output the same prediction category for two images with the same semantic category, that is, to fit the same learning goal, and the prediction of the classification under overfitting is very similar for multiple dissimilar images). This semantic overfitting will make other networks more prone to overfitting, that is, the features of two images with the same semantic category (referring to non-semantic features / semantic features) are too similar (or even the same), resulting in no distinction between the features of different images. Therefore, in an embodiment of the present application, an asynchronous training rate is adopted, that is, the training rate of the semantic classification network is higher than that of other networks that need to be trained in the image processing model, wherein the training rate is the update rate of the weight parameters, also known as the learning rate. In this way, it can be ensured that the weight parameters of other networks are updated more slowly than the weight parameters of the semantic classification network, effectively avoiding overfitting, thereby improving the training effect of the image processing model.

[0141] like Figure 3C As shown, the image processing model includes a shared initial feature extraction network, which can support the sharing of underlying features (i.e., initial features) and improve the training efficiency of the image processing model; the network to be trained can be selected according to the needs in the actual application scenario, which can improve the flexibility of training.

[0142] In some embodiments, see Figure 3D , Figure 3D is a flowchart of an artificial intelligence-based image processing method provided in an embodiment of the present application, which can be executed by an electronic device. Figure 3BBefore step 201 shown, in step 401, each of the multiple training videos may be storyboarded to obtain multiple storyboard segments, respectively; wherein the storyboard segments include multiple similar training images.

[0143] In the embodiment of the present application, the training image group can be generated manually (for example, some training images are manually selected to generate the training image group) or automatically. Here, an example method of automatic generation will be described.

[0144] For each of the multiple training videos, the training video is storyboarded to obtain multiple storyboard segments, wherein the storyboard segments include multiple similar training images, and the storyboard processing can be implemented with the help of open source tools. After obtaining the multiple storyboard segments, the multiple storyboard segments can be further screened, for example, the multiple storyboard segments of the training video are uniformly sampled at specific intervals (such as 1 minute), and step 402 is performed according to the sampled storyboard segments.

[0145] In step 402, a plurality of storyboard segments are sampled to obtain a plurality of training image groups; wherein the training image groups include at least one of the following: a plurality of training images belonging to the same storyboard segment; a plurality of training images belonging to different training videos.

[0146] For example, sampling processing (training image sampling processing) is performed on all storyboard segments obtained through step 401 to obtain multiple training image groups, wherein each training image group includes at least one of the following: multiple training images belonging to the same storyboard segment (the similarity relationship between the multiple training images is similar); multiple training images belonging to different training videos respectively (the similarity relationship between the multiple training images is dissimilar).

[0147] For example, the acquired training videos include training video V1 and training video V2, and the training video V1 is storyboarded to obtain storyboard segment V 11 and storyboard clip V 12 , among which, storyboard segment V 11 Include training images V that are similar to each other 111 and training images V 112 , Storyboard V 12 Include training images V that are similar to each other 121 and training images V 122 ; Perform storyboard processing on the training video V2 to obtain storyboard segment V 21 and storyboard clip V 22 , among which, storyboard segment V 21 Include training images V that are similar to each other 211 and training images V 212 , Storyboard V 22Include training images V that are similar to each other 221 and training images V 222 The sampling process in step 402 may refer to sampling all training images (including V 111 、V 112 、V 121 、V 122 、V 211 、V 212 、V 221 and V 222 ) to establish a training image group. For multiple training images belonging to the same storyboard, some examples are provided, such as training image V 111 and V 112 belongs to the same storyboard segment V 11 , training image V 121 and V 122 belongs to the same storyboard segment V 12 Some examples are also provided for multiple training images belonging to different training videos, such as training image V 111 and V 211 They belong to different training videos and training images V 121 and V 222 They belong to different training videos.

[0148] The sampling processing strategy in step 402 may be adjusted accordingly according to the number of training images required to be included in the training image group and the similarity relationship between the training images. For example, the following situations may exist.

[0149] 1) The training image group is a binary group, which includes two types. One is that the two training images in the binary group are similar. For this purpose, two training images can be randomly sampled from the same storyboard of a training video to establish a binary group; the other is that the two training images in the binary group are not similar. For this purpose, one training image can be randomly sampled from two storyboards from different training videos to establish a binary group. It is worth noting that the loss function corresponding to the binary group for metric learning can be a contrastive loss.

[0150] 2) The training image group is a triplet, which includes a first training image, a second training image similar to the first training image, and a third training image different from the first training image. To this end, two training images can be randomly sampled from the same storyboard segment of a training video and used as the first training image and the second training image respectively; a training image can be randomly sampled from any storyboard segment of another training video to serve as the third training image.

[0151] 3) The training image group is a quadruple, which includes a first training image, a second training image similar to the first training image, a third training image dissimilar to the first training image, and a fourth training image dissimilar to the first training image. In this regard, two training images can be randomly sampled from the same storyboard of a training video and used as the first training image and the second training image respectively; a training image can be randomly sampled from any storyboard of another training video as the third training image; and a training image can be randomly sampled from any storyboard of another training video as the fourth training image. The first training image, the third training image, and the fourth training image correspond to different training videos, respectively. It is worth noting that the loss function corresponding to the quadruple for metric learning can be a quadruplet loss (Quadruplet Loss).

[0152] like Figure 3D As shown, the embodiment of the present application can realize the automatic generation of training image groups, obtain high-quality training data, and save labor costs.

[0153] In some embodiments, see Figure 3E , Figure 3E is a flowchart of an artificial intelligence-based image processing method provided in an embodiment of the present application, which can be executed by an electronic device. Figure 3A After step 102 shown, in step 501, a correspondence between the sample video where the sample image is located and the target index can be established based on the correspondence between the sample image and the target index; wherein the target index includes any one of a non-semantic index and a semantic index.

[0154] In addition to supporting similar image queries, the embodiments of the present application can also support similar video queries. For example, the sample image is obtained by sampling the sample video, and the query image is obtained by sampling the query video. In this case, for each sample image, a correspondence between the sample video where the sample image is located and the target index can be established based on the correspondence between the sample image and the target index; wherein the target index includes any one of a non-semantic index and a semantic index. In this way, the correspondence between the sample video and the non-semantic index, and the correspondence between the sample video and the semantic index can be obtained.

[0155] exist Figure 3E middle, Figure 3A After step 105 shown, in step 502, the sample video where the similar sample image is located can be used as a similar sample video of the to-be-queried video where the to-be-queried image is located.

[0156] Here, after obtaining similar sample images of the to-be-queried image, the sample video where the similar sample images are located can be used as the similar sample video of the to-be-queried video where the to-be-queried image is located, that is, similar video query is implemented.

[0157] In step 503, a correspondence between the video to be queried and the similar target index obtained by similar query processing is established; wherein the similar target index includes any one of a similar non-semantic index and a similar semantic index.

[0158] Here, a correspondence relationship between the video to be queried and the similar non-semantic index obtained by similar query processing, and a correspondence relationship between the video to be queried and the similar semantic index obtained by similar query processing are established.

[0159] In step 504, the plurality of similar sample videos are sorted according to the correspondence between the similar sample videos and the target index, and the correspondence between the to-be-queried video and the similar target index.

[0160] For each similar sample video, the ranking score of the similar sample video is determined according to the corresponding relationship between the similar sample video and the target index, and the corresponding relationship between the query video and the similar target index. Then, the multiple similar sample videos are sorted according to the ranking scores, for example, in descending order of the ranking scores, so that the query requirements of the user can be better met.

[0161] In some embodiments, the above-mentioned sorting of multiple similar sample videos based on the correspondence between similar sample videos and target indexes, and the correspondence between videos to be queried and similar target indexes can be achieved in the following manner: performing any one of the following processing: sorting multiple similar sample videos based on the intersection-and-union ratio between semantic indexes corresponding to similar sample videos and similar semantic indexes corresponding to videos to be queried; fusing the intersection-and-union ratio between non-semantic indexes corresponding to similar sample videos and similar non-semantic indexes corresponding to videos to be queried, and the intersection-and-union ratio between semantic indexes corresponding to similar sample videos and similar semantic indexes corresponding to videos to be queried, and sorting multiple similar sample videos based on the obtained fused intersection-and-union ratio.

[0162] In an embodiment of the present application, sorting can be implemented based on semantic indexes. For example, the intersection-and-union ratio between the semantic index corresponding to the similar sample video and the similar semantic index corresponding to the query video is determined, the intersection-and-union ratio is used as the semantic sorting score, and multiple similar sample videos are sorted based on the semantic sorting score. The larger the semantic sorting score, the more semantically similar the corresponding similar sample video is to the query video, and therefore the similar sample video is ranked higher.

[0163] In an embodiment of the present application, a sorting process can be implemented by combining non-semantic indexes and semantic indexes. For example, the intersection-and-union ratio between the non-semantic index corresponding to the similar sample video and the similar non-semantic index corresponding to the video to be queried, and the intersection-and-union ratio between the semantic index corresponding to the similar sample video and the similar semantic index corresponding to the video to be queried are fused, and the obtained fused intersection-and-union ratio is used as a fused sorting score, and multiple similar sample videos are sorted according to the fused sorting score. The larger the fused sorting score, the more similar the corresponding similar sample video is to the video to be queried in terms of comprehensive degree (combined non-semantic and semantic), and therefore the similar sample video is ranked higher.

[0164] Through the above methods, the flexibility of sorting processing can be improved, and any of the above methods can be applied according to the needs of actual application scenarios.

[0165] like Figure 3E As shown, the embodiment of the present application can support similar video queries, and can also sort multiple queried videos in combination with semantics, which can further improve the query effect.

[0166] The following will describe an exemplary application of the embodiment of the present application in an actual application scenario. The embodiment of the present application can be applied to a scenario of similar image query or similar video query. As an example, the following is provided: Figure 4 The schematic diagram of bucket query (bucket retrieval) shown in FIG. 1 , where a bucket corresponds to a cluster, such as Figure 4 The classes 1, 2, ..., and Z are shown, where Z is an integer greater than 1. Figure 4 In the process, feature extraction processing can be performed on multiple sample images in the image library respectively, and then the image features of the multiple sample images are clustered to obtain multiple clusters and the cluster center (i.e., cluster center) of each cluster, where the image features are as follows: Figure 4 As shown in (41, 98, ..., 61), (29, 81, ..., 79), etc., the cluster centers are as follows Figure 4 The center of class 1, the center of class 2, etc. are shown, and each cluster center corresponds to multiple sample images. For the query image, feature extraction processing is also performed on it to obtain image features, and then the image features of the query image are compared with multiple cluster centers. According to the comparison results, similar sample images of the query image are screened out from the image library. For example, the sample image corresponding to the cluster center closest to the image feature of the query image can be used as a similar sample image. On this basis, the sample video where the similar sample image is located can also be used as a similar sample video of the query video where the query image is located.

[0167] It is worth noting that when determining the cluster center that is closest to the image features of the image to be queried, the image features of the sample image corresponding to the cluster center can also be compared with the image features of the image to be queried. For example, when the distance between the image features of any sample image corresponding to the cluster center and the image features of the image to be queried is less than a set distance threshold, the any sample image is taken as a similar sample image. In this way, further screening of the sample images can be achieved.

[0168] In addition, after determining the sample video where the similar sample image is located, the sample video can also be compared with the image features of the video to be queried frame by frame. For example, determine the distance between the image features of the first frame of the sample video and the image features of the first frame of the video to be queried, the distance between the image features of the second frame of the sample video and the image features of the second frame of the video to be queried... When the distance obtained after comparing the image features of each frame of the sample video and the video to be queried is less than the distance threshold, the sample video is regarded as a similar sample video, so that further screening of the sample videos can be achieved.

[0169] For feature extraction processing, in the solutions provided by the related art, the model is usually trained by using similarity (distance) metric learning, and the image is subjected to feature extraction processing based on the trained model to obtain image features. However, the similarity metric learning method indiscriminately focuses on all features in the image, resulting in at least the following problems in the solutions provided by the related art: 1) It is impossible to recall sample images that are similar to the foreground of the image to be queried and dissimilar to the background. For example, in some application scenarios, the attacker replaces the background in image A by cutting out the image to obtain image B. If the solutions provided by the related art are used, image A and image B will be mistakenly identified as dissimilar; 2) It is impossible to prioritize the recall results (similar sample images / similar sample videos). For example, if the video to be queried includes a dog, some of the recall results include dogs, and the other part of the recall results include wolves, then the recall results including dogs should be ranked before the recall results including wolves when sorting. However, this cannot be achieved according to the solutions provided by the related art.

[0170] In view of this, the embodiment of the present application provides a dual index query solution. As an example, the following is provided: Figure 5 Schematic diagram of the image processing method based on artificial intelligence. Figure 5In the example, if multiple sample images in the image library are clustered according to the solution provided by the related art, insufficient semantic similarity may occur in the obtained clusters. For example, multiple sample images in cluster 1 include dogs, sheep, horses, and cows; multiple sample images in cluster 2 include dogs, sheep, horses, and airplanes; and multiple sample images in cluster N include airplanes and vehicles, where N is an integer greater than 1.

[0171] In the embodiment of the present application, Figure 5 As shown, for each sample image, the initial feature extraction network is first used to extract the initial features to obtain the initial features, and then the non-semantic feature extraction network is used to extract the non-semantic features of the initial features to obtain the non-semantic features of the sample image; at the same time, the semantic feature extraction network is used to extract the semantic features of the initial features to obtain the semantic features of the sample image. Then, clustering is performed according to the non-semantic features corresponding to the multiple sample images to obtain the non-semantic index, and an index system 1 based on the non-semantic index is established; clustering is performed according to the semantic features corresponding to the multiple sample images to obtain the semantic index, and an index system 2 based on the semantic index is established. When the image to be queried is obtained, similar query processing is performed on the index system 1, that is, non-semantic recall is performed; at the same time, similar query processing is performed on the index system 2, that is, semantic recall is performed. Finally, the recall result of the non-semantic recall is merged with the recall result of the semantic recall, and the merged recall result is sorted according to the semantic index (or according to the semantic index and the non-semantic index).

[0172] For ease of understanding, the following text will take the scenario of similar video query as an example to illustrate the artificial intelligence-based image processing method provided in the embodiment of the present application in the form of steps.

[0173] 1) Train the image processing model. As an example, the following is provided: Figure 6 The image processing model shown in the figure mainly includes three modules, namely the initial feature extraction network, the non-semantic branch network and the semantic branch network. The initial feature extraction network can be a convolutional neural network (CNN), such as ResNet101, and the network architecture is as follows:

[0174]

[0175]

[0176] Among them, the "7×7" in "7×7,64,stride 2" represents the convolution kernel size, "64" represents the number of output channels, and "stride2" represents the stride of 2; in addition, "max pool" represents the maximum pooling layer, and "×3blocks" represents the existence of 3 identical structures, and the rest of the parameters are similar.

[0177] The non-semantic branch network includes a non-semantic feature extraction network, which is trained using metric learning. Its network architecture can be shown as follows:

[0178]

[0179] Among them, 128 represents the dimension of the non-semantic features output by the non-semantic feature extraction network.

[0180] The semantic branch network includes a semantic feature extraction network and a semantic classification network. The network architecture of the semantic branch network can be shown as follows:

[0181]

[0182] Among them, Pool_cr2 and Embedding_class constitute a semantic feature extraction network, and Fc_cr constitutes a semantic classification network; 64 represents the dimension of the semantic features output by the semantic feature extraction network; Nclass represents the number of semantic categories to be learned, for example, when the ImageNet image set is used for training, Nclass can be 1000. It is worth noting that the above network architecture is only an example and does not constitute a limitation on the embodiments of the present application.

[0183] The process of training an image processing model mainly involves the following steps.

[0184] ① Parameter initialization. For the initial feature extraction network, it can be trained through open source image sets (such as ImageNet image set, Open Image image set, etc.) to complete the initialization of weight parameters. For non-semantic branch networks and semantic branch networks, the weight parameters can be randomly initialized, for example, a Gaussian distribution with a variance of 0.01 and a mean of 0 can be used for initialization.

[0185] ② Determine the weight parameters that need to be learned (i.e., need to be updated). Here, the first way is to determine the initial feature extraction network, the non-semantic branch network, and the semantic branch network as networks that need to be trained; the second way is to determine the non-semantic branch network and the semantic branch network as networks that need to be trained, while the initial feature extraction network does not need to be trained. Among them, for each network that needs to be trained, the weight parameters within it need to be learned.

[0186] ③ Determine the learning rate (Learning Rate), which corresponds to the training rate above. In the embodiment of the present application, compared with other networks in the image processing model, the semantic classification network is more prone to overfitting (because the task of semantic classification is to output the same prediction category for two images with the same semantic category, that is, to fit the same learning goal, and the prediction of classification under overfitting is very similar for multiple dissimilar images). The prediction category is very similar), and this semantic overfitting will make other networks more prone to overfitting, that is, the features of two images with the same semantic category (referring to non-semantic features / semantic features) are too similar (or even the same), resulting in no distinction between the features of different images. In the embodiment of the present application, in order to avoid this situation, an asynchronous learning rate can be used, that is, the learning rate of the semantic classification network is higher than the learning rate of other networks, so that the update rate of the weight parameters of other networks can be guaranteed to be slower than that of the semantic classification network, effectively preventing other networks from overfitting. For example, the learning rate of the semantic classification network can be set to 0.005, and the learning rates of other networks can be uniformly set to 0.0005, so that the update rate of the weight parameters of other networks is 1 / 10 of that of the semantic classification network.

[0187] ④ Learning process (training process). Here, the image processing model can be iterated by epoch rounds. Epoch is an integer greater than 1 and can be set according to the actual application scenario. In each round of iteration, all training images are processed once. For example, all training images can be divided according to batch-size to obtain Nb batches. The following processing is performed for each batch. Batch-size can be set according to the actual application scenario.

[0188] a) Forward propagation processing. Here, each training image in the batch is forward propagated through the image processing model. For the sake of distinction, the non-semantic features output by the non-semantic branch network are named em1, the semantic features output by the semantic feature extraction network are named em2, and the predicted classification features output by the semantic classification network are named fc2.

[0189] b) Loss calculation. For em1, find the corresponding triplet in the batch (corresponding to the training image group above) and calculate triplet loss1 (corresponding to the non-semantic feature loss value above); for em2, similar to em1, find the corresponding triplet in the batch and calculate triplet loss2 (corresponding to the semantic feature loss value above); for fc2, combine the labeled classification features to calculate kl loss (corresponding to the semantic classification loss value above), which is also called distribution similarity loss or distribution entropy loss. Then, triplet loss1, triplet loss2 and kl loss are fused to obtain the fused loss value.

[0190] c) Weight parameter update: For example, a stochastic gradient descent algorithm or other gradient descent algorithms may be used to perform back propagation processing on the obtained fusion loss value, and in the process of back propagation, the weight parameters of the network to be trained are updated along the gradient descent direction.

[0191] The training process of the image processing model also involves the preparation of training data, which is explained below with examples. In an embodiment of the present application, triplets can be used as training data for the image processing model. The form of the triplets is (anchor, positive, negative), where the anchor (corresponding to the first training image above) and the positive (corresponding to the second training image above) are similar (or the same), and the anchor and the negative (corresponding to the third training image above) are not similar. It is difficult to extract triplets from a large number (such as millions or even tens of millions) of training images. Therefore, an embodiment of the present application provides an effective and fast way for similar video queries, as shown below.

[0192] ① Perform storyboard processing on V training videos, so that multiple storyboard segments can be obtained for each training video, and each storyboard segment includes multiple similar training images, where V is an integer greater than 1. The embodiment of the present application does not limit the method of storyboard processing, for example, it can be implemented by the PySceneDetect tool. As an example, the following is provided: Figure 7 The schematic diagram of the storyboard segments shown in FIG. 1 provides five example storyboard segments. Taking storyboard segment 71 as an example, the training images 711 and 712 therein are obtained by randomly sampling in storyboard segment 71. Figure 7 It can be seen that training image 711 is similar to training image 712.

[0193] ② For each training video, evenly select M storyboard segments from the obtained multiple storyboard segments, where M is an integer greater than 1. For example, when M=5, for a training video with a duration of 5 minutes, a storyboard segment is taken every 1 minute. For each storyboard segment taken out, randomly sample 2 training images in the storyboard segment to form an image pair, that is, obtain the (anchor, positive) in the triplet.

[0194] In this way, for all V training videos, V*M image pairs can be obtained, and the subsequent sampling in training is performed at the granularity of image pairs.

[0195] ③ In the training of each batch, any image pair i in the batch-size image pairs is denoted as (ai, pi) and the following processing is performed.

[0196] a) Randomly select an image pair from a different training video than image pair i from other image pairs in this batch, and use a random training image in the selected image pair as the negative in the triplet.

[0197] b) Repeat step a) five times to obtain five triplets, namely (ai, pi, ni1), (ai, pi, ni2), (ai, pi, ni3), (ai, pi, ni4) and (ai, pi, ni5).

[0198] c) Calculate the triplet loss1 for the above five triplets respectively, and perform fusion processing to obtain the non-semantic feature loss value L of image pair i triplet1 In the embodiment of the present application, the fusion process may be direct summation or weighted summation, etc., which is not limited. For example, the formula for calculating triplet loss1 according to the triplet (ai, pi, ni1) is as follows:

[0199] triplet loss1=max(‖ai(em1)-pi(em1)‖-‖ai(em1)-ni1(em1)‖+α1,0)

[0200] Among them, ai(em1) represents the non-semantic feature of the training image ai in the triplet, and so on; ‖ai(em1)-pi(em1)‖ represents the distance between the non-semantic feature of the training image ai and the non-semantic feature of the training image pi (such as L2 distance), and so on; α1 is a margin parameter, which can be set to 0.4. Here, the value of α1 can be adjusted according to the dimension of the non-semantic feature, for example, it can be positively correlated with the dimension of the non-semantic feature. The purpose of calculating triplet loss1 is to constrain the value obtained by subtracting the non-semantic feature distance between ai and ni1 (i.e., the distance corresponding to the non-semantic feature) from the non-semantic feature distance between ai and pi to be greater than α1.

[0201] d) Calculate the triplet loss2 for the above five triplets respectively, and perform fusion processing to obtain the non-semantic feature loss value L of image pair i triplet2 Similarly, for example, the formula for calculating triplet loss2 based on the triplet (ai, pi, ni1) is as follows:

[0202] triplet loss2=max(‖ai(em2)-pi(em2)‖-‖ai(em2)-ni1(em2)‖+α2,0)

[0203] Here, the value of α2 can be adjusted according to the dimension of the semantic feature, for example, it can be positively correlated with the dimension of the semantic feature. α2 can be the same as or different from α1.

[0204] e) It is worth noting that for the above-generated image pairs, i.e., a total of V*M*2 images, semantic classification processing can be performed according to the semantic classification model to obtain the labeled classification features of each image. The semantic classification model is, for example, a ResNet50 model trained according to an open source image set, and the labeled classification features are, for example, a 1*1000 prediction probability vector.

[0205] In this way, the kl loss can be calculated for the above five triplets respectively, and the fusion processing can be performed to obtain the semantic classification loss value L of image pair i kl For example, the formula for calculating the kl loss based on a triple is as follows:

[0206]

[0207] Among them, x r represents the rth training image in the triplet, R represents the number of training images in the triplet, which is 3; x r (fc1) represents the labeled classification feature of the rth training image in the triplet, x r(fc2) represents the predicted classification feature of the rth training image in the triple. According to the above formula, it can be determined that when x r (fc1) = x r (fc2), The value of is the smallest, which is 0.

[0208] f) For the image pair i, the non-semantic feature loss value, semantic feature loss value and semantic classification loss value obtained above can be fused to obtain a fusion loss value. For example, the fusion loss value L total The calculation formula is as follows:

[0209] L total =w1L triplet1 +w2L triplet2 +w3L kl

[0210] Among them, w1, w2 and w3 are all numbers greater than 0, and can be set according to actual application scenarios, such as being set to 1, 1 and 0.1 respectively.

[0211] 2) Extract the non-semantic features and semantic features of the sample image according to the trained image processing model. Here, for each of the S sample videos, uniform sampling is performed to obtain P sample images, so that a total of P*S sample images can be obtained, wherein the sampling frequency of the uniform sampling process here is not limited, for example, it can be five frames per second. It is worth noting that the sample video can be the above-mentioned training video, or it can be a new video different from the training video.

[0212] For each sample image obtained, forward propagation processing is performed through the trained image processing model to obtain non-semantic features output by the non-semantic feature extraction network and semantic features output by the semantic feature extraction network.

[0213] 3) Cluster multiple non-semantic features to obtain non-semantic indexes, and cluster multiple semantic features to obtain semantic indexes. For example, if the number of sample images is 10 million, the non-semantic features corresponding to all sample images can be clustered to obtain 100,000 non-semantic cluster centers (i.e., non-semantic indexes), and index system 1, i.e., Center 1, is established based on these 100,000 non-semantic cluster centers; at the same time, the semantic features corresponding to all sample images can be clustered to obtain 100,000 semantic cluster centers (i.e., semantic indexes), and index system 2, i.e., Center 2, is established based on these 100,000 semantic cluster centers.

[0214] The embodiments of the present application do not limit the clustering algorithm used for clustering processing. For example, it can be the Kmeans algorithm, hierarchical clustering algorithm, K-means++ algorithm, etc. For example, the K-means++ algorithm selects K clustering centers in the following way: Assume that n clustering centers have been selected (0 < n < K). Then, when selecting the (n + 1)-th clustering center, the points (i.e., indices) farther away from the current n clustering centers will have a higher probability of being selected as the (n + 1)-th clustering center. Among them, the first clustering center can be selected by random selection. The embodiments of the present application do not limit the number of non-semantic clustering centers and the number of semantic clustering centers.

[0215] 4) Establish an inverted list (also known as an inverted matrix). On the basis of obtaining the non-semantic index and the semantic index, an non-semantic inverted list corresponding to the non-semantic index and a semantic inverted list corresponding to the semantic index can be further established. Among them, the non-semantic inverted list is used to record the correspondence between the non-semantic index and the sample image, and the semantic inverted list is used to record the correspondence between the semantic index and the sample image.

[0216] For the sake of easy understanding, the process of establishing the semantic inverted list is taken as an example for illustration.

[0217] ① Initialize the values in the two-dimensional semantic inverted list to all 0. The dimension of this semantic inverted list is (K, S), where K represents the number of semantic indices, for example, it can be 100,000; S represents the number of sample videos. A certain value (k, s, x) in the semantic inverted list means that in the s-th sample video, the semantic indices corresponding to x sample images (i.e., there is a correspondence) are all the k-th semantic index.

[0218] It should be noted that for the semantic index, a sample video-semantic index list VC can also be initialized. The length of this list is S. After initialization, VC[1] = [], VC[2] = [], …, VC[S] = [], that is, it is initialized to be empty. Among them, VC[s] represents which semantic indices the s-th sample video has a correspondence with. For example, VC[1] = [1, 3, 4, 10] means that the first sample video has a correspondence with the first semantic index, the third semantic index, the fourth semantic index, and the tenth semantic index.

[0219] ② For each sample video (taking the s-th sample video as an example), traverse multiple sample images in the sample video, and perform the following processing on the traversed sample images. Among them, the sample images in the sample video here can be obtained by sampling processing in the above step 2), or can be obtained by re-sampling processing according to a new sampling frequency.

[0220] a) Determine the semantic index j that is closest to the semantic feature of the traversed sample image.

[0221] b) updating the semantic inverted list according to the semantic index j, that is, updating the value (j, s, x) in the semantic inverted list to (j, s, x+1).

[0222] c) Update the list VC according to the semantic index j, that is, add the semantic index j to VC[s] in the list VC.

[0223] Similar to the process of establishing a semantic inverted list, a non-semantic inverted list can also be established. It is worth noting that in the process of establishing a non-semantic inverted list, a sample video-non-semantic index list can be established, or a sample video-non-semantic index list can not be established, depending on the actual application scenario.

[0224] 5) Perform video recall according to the inverted list. Here, the process of video recall is explained in steps.

[0225] ① When the query video is obtained, the query video is uniformly sampled (for example, the sampling frequency is five frames per second) to obtain multiple query images. For each query image, the trained image processing model is used for forward propagation to obtain the non-semantic features output by the non-semantic feature extraction network in the image processing model and the semantic features output by the semantic feature extraction network.

[0226] ② Initialize the query video-semantic index vector Vq = [], that is, initialize it to empty, and Vq is used to represent the corresponding relationship between the query video and the semantic index.

[0227] ③ Traverse multiple query images, and perform video recall based on the non-semantic features and semantic features of the traversed query images. For ease of understanding, the video recall based on the semantic features of the traversed query images is taken as an example for explanation.

[0228] a) Determine the semantic index t that is closest to the semantic feature of the traversed query image, and add the semantic index t to the vector Vq.

[0229] b) According to the semantic index t, obtain the t-th vector Vt in the semantic inverted list, where the dimension of the vector Vt is 1*S. When the s-th value in the vector Vt is x, it means that there is a corresponding relationship between the x sample images in the s-th sample video and the semantic index t.

[0230] c) Traverse multiple values ​​in the vector Vt. When the traversed value is equal to 0, skip the traversed value and continue to traverse the next value; when the traversed value is greater than 0, add the sample video corresponding to the traversed value to the semantic recall video set.

[0231] Similarly, a non-semantic recall video set can be obtained based on the non-semantic features of the query image.

[0232] ④ The semantic recall video set and the non-semantic recall video set are merged to obtain a final recall video set Set, which includes Ns sample videos, and these Ns sample videos correspond to the similar sample videos above, where Ns is an integer greater than 0.

[0233] ⑤ Sort the Ns sample videos in the recalled video set Set, as shown in the following example.

[0234] a) Initialize a vector of all zeros of length Ns, denoted as Rank.

[0235] b) Traverse the Ns sample videos in the recalled video set Set. For ease of understanding, taking the traversed sample video as the sth sample video as an example, first determine the vector VC[s] corresponding to the sth sample video from the list VC. Then, determine the intersection between the vector Vq and the vector VC[s], and determine the number overlap of semantic indexes included in the intersection, where each semantic index in the intersection exists in both the vector Vq and the vector VC[s]; at the same time, determine the union between the vector Vq and the vector VC[s], and determine the number union of semantic indexes included in the union, where each semantic index in the union exists in the vector Vq or the vector VC[s]. Calculate Rank[s] = overlap / union, where Rank[s] is the semantic ranking score of the sth sample video. The larger the semantic ranking score, the more semantically similar it is to the query video.

[0236] It is worth noting that the above method calculates the ranking score based on the semantic index. In some embodiments, the ranking score can also be calculated in combination with the non-semantic index and the semantic index. For example, the fused ranking score Score[s] of the s-th sample video = a*Rank1[s]+b*Rank2[s], where Rank1[s] represents the non-semantic ranking score of the s-th sample video determined according to the sample video-non-semantic index list, and Rank2[s] represents the semantic ranking score of the s-th sample video determined according to the sample video-semantic index list; a and b represent weights, which can be set according to the actual application scenario. For example, when the sum of a and b is 1, a and b can both be 0.5. Among them, the fused ranking score corresponds to the fused intersection-union ratio mentioned above.

[0237] c) Sorting the Ns sample videos in the recalled video set Set according to the order of the ranking scores (referring to the semantic ranking scores or the fusion ranking scores) from large to small.

[0238] Through the embodiments of the present application, at least the following technical effects can be achieved: 1) semantic query capability is provided, which can improve the accuracy of video recall and make up for the deficiency of non-semantic recall compared with the solutions provided by related technologies; 2) multiple sample videos recalled are sorted in combination with semantic indexing, which can improve the TopK hit rate of recall, wherein the TopK hit rate is used to measure the recall effect. For example, among the recalled Top100 sample videos, there are actually 60 similar videos, and the Top100 hit rate is 60%; 3) through multi-task learning that shares the initial feature extraction network, non-semantic features and semantic features can be learned simultaneously under one model framework, thereby improving learning efficiency (training efficiency); 4) by obtaining labeled classification features through a pre-trained semantic classification model, the cost of manual labeling can be saved.

[0239] The following is a description of an exemplary structure of an artificial intelligence-based image processing device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the artificial intelligence-based image processing device 455 stored in the memory 450 may include: a feature extraction module 4551, which is used to perform feature extraction processing on the sample image to obtain the non-semantic features and semantic features of the sample image; an index establishment module 4552, which is used to establish a non-semantic index according to the non-semantic features corresponding to multiple sample images, and to establish a semantic index according to the semantic features corresponding to multiple sample images; the feature extraction module 4551 is also used to perform feature extraction processing on the query image to obtain the non-semantic features and semantic features of the query image; a similarity query module 4553 is used to perform similarity query processing on the non-semantic index according to the non-semantic features of the query image to obtain a similar non-semantic index, and to perform similarity query processing on the semantic index according to the semantic features of the query image to obtain a similar semantic index; the similarity query module 4553 is also used to use the sample image corresponding to the similar non-semantic index, and at least part of the sample images in the sample image corresponding to the similar semantic index as similar sample images of the query image.

[0240] In some embodiments, the artificial intelligence-based image processing device 455 also includes a training module, which is used to: perform feature extraction processing on the training images in the training image group through the image processing model to obtain non-semantic features and semantic features of the training images, and perform semantic classification processing on the semantic features of the training images to obtain predicted classification features; perform loss calculation processing according to the non-semantic features, semantic features and predicted classification features of the training images, respectively, to obtain non-semantic feature loss values, semantic feature loss values ​​and semantic classification loss values ​​of the corresponding training image group; perform fusion processing on the non-semantic feature loss values, semantic feature loss values ​​and semantic classification loss values ​​of the training image group, and train the image processing model according to the obtained fusion loss values; wherein the trained image processing model is used to perform feature extraction processing on sample images and images to be queried.

[0241] In some embodiments, the image processing model includes an initial feature extraction network, a non-semantic feature extraction network, a semantic feature extraction network and a semantic classification network; the training module is also used to: perform initial feature extraction processing on the training image through the initial feature extraction network to obtain the initial features of the training image; perform non-semantic feature extraction processing on the initial features of the training image through the non-semantic feature extraction network to obtain the non-semantic features of the training image; perform semantic feature extraction processing on the initial features of the training image through the semantic feature extraction network to obtain the semantic features of the training image; perform semantic classification processing on the semantic features of the training image through the semantic classification network to obtain the predicted classification features of the training image.

[0242] In some embodiments, the training module is also used to perform any of the following processing: training a non-semantic feature extraction network, a semantic feature extraction network, and a semantic classification network according to a fusion loss value; training an initial feature extraction network, a non-semantic feature extraction network, a semantic feature extraction network, and a semantic classification network according to a fusion loss value; wherein the training rate of the semantic classification network is higher than that of the network in the image processing model that is different from the semantic classification network.

[0243] In some embodiments, the training image group includes a first training image, a second training image similar to the first training image, and a third training image dissimilar to the first training image; the training module is further used to: determine the non-semantic feature loss value of the training image group based on the non-semantic distance between the non-semantic features of the first training image and the non-semantic features of the second training image, and the non-semantic distance between the non-semantic features of the first training image and the non-semantic features of the third training image; determine the semantic feature loss value of the training image group based on the semantic distance between the semantic features of the first training image and the semantic features of the second training image, and the semantic distance between the semantic features of the first training image and the semantic features of the third training image; determine the semantic classification loss value of any training image in the training image group based on the labeled classification features and predicted classification features of any training image, and fuse the semantic classification loss values ​​corresponding to the first training image, the second training image and the third training image, respectively, to obtain the semantic classification loss value of the training image group.

[0244] In some embodiments, the training module is also used to: perform any of the following processing: obtain the labeled classification features set for any training image; perform semantic classification processing on any training image through a semantic classification model to obtain the labeled classification features; wherein the semantic classification model is different from the image processing model.

[0245] In some embodiments, the training module is further used to: perform storyboard processing on each of a plurality of training videos to obtain a plurality of storyboard segments, respectively; wherein the storyboard segments include a plurality of similar training images; perform sampling processing on the plurality of storyboard segments to obtain a plurality of training image groups; wherein the training image groups include at least one of the following: a plurality of training images belonging to the same storyboard segment; a plurality of training images belonging to different training videos, respectively.

[0246] In some embodiments, the sample image is obtained by sampling the sample video, and the query image is obtained by sampling the query video; the similarity query module 4553 is also used to: use the sample video where the similar sample image is located as the similar sample video of the query video.

[0247] In some embodiments, the artificial intelligence-based image processing device 455 also includes a sorting module, which is used to: establish a correspondence between the sample video where the sample image is located and the target index according to the correspondence between the sample image and the target index; wherein the target index includes any one of a non-semantic index and a semantic index; establish a correspondence between the video to be queried and a similar target index obtained by similar query processing; wherein the similar target index includes any one of a similar non-semantic index and a similar semantic index; sort multiple similar sample videos according to the correspondence between similar sample videos and the target index, and the correspondence between the video to be queried and the similar target index.

[0248] In some embodiments, the sorting module is also used to perform any of the following processing: sorting multiple similar sample videos according to the intersection-and-union ratio between the semantic indexes corresponding to the similar sample videos and the similar semantic indexes corresponding to the video to be queried; fusing the intersection-and-union ratio between the non-semantic indexes corresponding to the similar sample videos and the similar non-semantic indexes corresponding to the video to be queried, and the intersection-and-union ratio between the semantic indexes corresponding to the similar sample videos and the similar semantic indexes corresponding to the video to be queried, and sorting multiple similar sample videos according to the obtained fused intersection-and-union ratio.

[0249] In some embodiments, the index establishment module 4552 is also used to: cluster the non-semantic features corresponding to multiple sample images, and use the obtained multiple non-semantic cluster centers as non-semantic indexes; cluster the semantic features corresponding to multiple sample images, and use the obtained multiple semantic cluster centers as semantic indexes; determine the non-semantic distances between the non-semantic features of the sample images and the multiple non-semantic indexes, and establish a correspondence between the sample images and the non-semantic indexes that meet the non-semantic distance conditions; determine the semantic distances between the semantic features of the sample images and the multiple semantic indexes, and establish a correspondence between the sample images and the semantic indexes that meet the semantic distance conditions.

[0250] In some embodiments, the similarity query module 4553 is also used to: respectively determine the non-semantic distances between the non-semantic features of the image to be queried and multiple non-semantic indexes, and use the non-semantic indexes that meet the non-semantic distance conditions as similar non-semantic indexes; respectively determine the semantic distances between the semantic features of the image to be queried and multiple semantic indexes, and use the semantic indexes that meet the semantic distance conditions as similar semantic indexes.

[0251] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions (i.e., executable instructions), which are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the above-mentioned artificial intelligence-based image processing method in the embodiment of the present application.

[0252] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E The image processing method based on artificial intelligence is shown.

[0253] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0254] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0255] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0256] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0257] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method based on artificial intelligence, characterized in that: The method comprises: Performing feature extraction processing on the sample image to obtain non-semantic features and semantic features of the sample image; Establishing a non-semantic index according to the non-semantic features respectively corresponding to the plurality of sample images, and establishing a semantic index according to the semantic features respectively corresponding to the plurality of sample images; Performing feature extraction processing on the query image to obtain non-semantic features and semantic features of the query image; Performing similarity query processing on the non-semantic index according to the non-semantic features of the image to be queried to obtain a similar non-semantic index, and performing similarity query processing on the semantic index according to the semantic features of the image to be queried to obtain a similar semantic index; The sample images corresponding to the similar non-semantic indexes and at least part of the sample images corresponding to the similar semantic indexes are used as similar sample images of the image to be queried.

2. The method according to claim 1, characterized in that Before performing feature extraction processing on the sample image, the method further includes: Performing feature extraction processing on the training images in the training image group through the image processing model to obtain non-semantic features and semantic features of the training images, and performing semantic classification processing on the semantic features of the training images to obtain predicted classification features; Perform loss calculation processing according to the non-semantic features, semantic features and predicted classification features of the training images, respectively, to obtain non-semantic feature loss values, semantic feature loss values ​​and semantic classification loss values ​​corresponding to the training image group; Performing fusion processing on the non-semantic feature loss value, the semantic feature loss value and the semantic classification loss value of the training image group, and training the image processing model according to the obtained fusion loss value; The trained image processing model is used to perform feature extraction processing on the sample image and the image to be queried.

3. The method according to claim 2, characterized in that The image processing model includes an initial feature extraction network, a non-semantic feature extraction network, a semantic feature extraction network and a semantic classification network; The step of performing feature extraction processing on the training images in the training image group by using the image processing model to obtain non-semantic features and semantic features of the training images includes: Performing initial feature extraction processing on the training image through the initial feature extraction network to obtain initial features of the training image; Performing non-semantic feature extraction processing on the initial features of the training image through the non-semantic feature extraction network to obtain the non-semantic features of the training image; Performing semantic feature extraction processing on the initial features of the training image through the semantic feature extraction network to obtain the semantic features of the training image; The performing semantic classification processing on the semantic features of the training image to obtain predicted classification features includes: The semantic classification network performs semantic classification processing on the semantic features of the training image to obtain predicted classification features of the training image.

4. The method according to claim 3, characterized in that The step of training the image processing model according to the obtained fusion loss value comprises: Perform any of the following: Training the non-semantic feature extraction network, the semantic feature extraction network, and the semantic classification network according to the fusion loss value; Training the initial feature extraction network, the non-semantic feature extraction network, the semantic feature extraction network, and the semantic classification network according to the fusion loss value; Among them, the training rate of the semantic classification network is higher than the network in the image processing model that is different from the semantic classification network.

5. The method according to claim 2, characterized in that: The training image group includes a first training image, a second training image similar to the first training image, and a third training image not similar to the first training image; The loss calculation process is performed according to the non-semantic features, semantic features and predicted classification features of the training images to obtain the non-semantic feature loss value, semantic feature loss value and semantic classification loss value corresponding to the training image group, including: Determining a non-semantic feature loss value of the training image group according to a non-semantic distance between the non-semantic feature of the first training image and the non-semantic feature of the second training image, and a non-semantic distance between the non-semantic feature of the first training image and the non-semantic feature of the third training image; Determining a semantic feature loss value of the training image group according to a semantic distance between a semantic feature of the first training image and a semantic feature of the second training image, and a semantic distance between a semantic feature of the first training image and a semantic feature of the third training image; According to the labeled classification features and predicted classification features of any one of the training images in the training image group, the semantic classification loss value of the any one of the training images is determined, and the semantic classification loss values ​​corresponding to the first training image, the second training image and the third training image are fused respectively to obtain the semantic classification loss value of the training image group.

6. The method according to claim 5, characterized in that Before determining the semantic classification loss value of any one of the training images in the training image group according to the annotated classification features and the predicted classification features of the training image, the method further includes: Perform any of the following: Obtaining a label classification feature set for any one of the training images; Perform semantic classification processing on any one of the training images through a semantic classification model to obtain labeled classification features; wherein the semantic classification model is different from the image processing model.

7. The method according to claim 2, characterized in that: Before performing feature extraction processing on the training images in the training image group by the image processing model, the method further includes: Performing storyboard processing on each of the plurality of training videos to obtain a plurality of storyboard segments, wherein the storyboard segments include a plurality of similar training images; Performing sampling processing on the plurality of storyboard segments to obtain a plurality of training image groups; The training image group includes at least one of the following: a plurality of training images belonging to the same storyboard segment; or a plurality of training images belonging to different training videos.

8. The method according to any one of claims 1 to 7, characterized in that: The sample image is obtained by sampling a sample video, and the image to be queried is obtained by sampling a video to be queried; After using the sample images corresponding to the similar non-semantic indexes and at least part of the sample images corresponding to the similar semantic indexes as similar sample images of the image to be queried, the method further includes: The sample video where the similar sample image is located is used as the similar sample video of the video to be queried.

9. The method according to claim 8, characterized in that The method further comprises: According to the correspondence between the sample image and the target index, a correspondence between the sample video where the sample image is located and the target index is established; wherein the target index includes any one of the non-semantic index and the semantic index; Establishing a correspondence between the video to be queried and the similar target index obtained by the similar query processing; wherein the similar target index includes any one of the similar non-semantic index and the similar semantic index; According to the correspondence between the similar sample videos and the target index, and the correspondence between the to-be-queried video and the similar target index, a plurality of the similar sample videos are sorted.

10. The method according to claim 9, characterized in that The sorting process of the plurality of similar sample videos according to the correspondence between the similar sample videos and the target index, and the correspondence between the to-be-queried video and the similar target index, comprises: Perform any of the following: Sorting the plurality of similar sample videos according to an intersection-and-union ratio between a semantic index corresponding to the similar sample video and a similar semantic index corresponding to the video to be queried; The intersection-and-union ratios of the non-semantic indexes corresponding to the similar sample videos and the similar non-semantic indexes corresponding to the video to be queried, as well as the intersection-and-union ratios of the semantic indexes corresponding to the similar sample videos and the similar semantic indexes corresponding to the video to be queried are fused, and the multiple similar sample videos are sorted according to the obtained fused intersection-and-union ratios.

11. The method according to any one of claims 1 to 7, characterized in that: The step of establishing a non-semantic index according to the non-semantic features respectively corresponding to the plurality of sample images, and establishing a semantic index according to the semantic features respectively corresponding to the plurality of sample images, comprises: Performing clustering processing on the non-semantic features respectively corresponding to the plurality of sample images, and using the obtained plurality of non-semantic cluster centers as non-semantic indexes; Performing clustering processing on the semantic features respectively corresponding to the plurality of sample images, and using the obtained plurality of semantic clustering centers as semantic indexes; The method further comprises: respectively determining the non-semantic distances between the non-semantic features of the sample image and the plurality of non-semantic indexes, and establishing a corresponding relationship between the sample image and the non-semantic indexes satisfying a non-semantic distance condition; Semantic distances between the semantic features of the sample image and the plurality of semantic indexes are determined respectively, and a corresponding relationship between the sample image and the semantic indexes satisfying a semantic distance condition is established.

12. The method according to any one of claims 1 to 7, characterized in that: The performing similarity query processing on the non-semantic index according to the non-semantic features of the image to be queried to obtain a similar non-semantic index includes: Determine the non-semantic distances between the non-semantic features of the image to be queried and the plurality of non-semantic indexes respectively, and use the non-semantic indexes that meet the non-semantic distance condition as similar non-semantic indexes; The performing similarity query processing on the semantic index according to the semantic features of the image to be queried to obtain a similar semantic index includes: Semantic distances between the semantic features of the image to be queried and the plurality of semantic indexes are determined respectively, and the semantic indexes satisfying the semantic distance condition are used as similar semantic indexes.

13. An image processing device based on artificial intelligence, characterized in that: The device comprises: A feature extraction module is used to perform feature extraction processing on the sample image to obtain non-semantic features and semantic features of the sample image; An index building module, used to build a non-semantic index according to the non-semantic features respectively corresponding to the plurality of sample images, and to build a semantic index according to the semantic features respectively corresponding to the plurality of sample images; The feature extraction module is further used to perform feature extraction processing on the query image to obtain non-semantic features and semantic features of the query image; A similarity query module, used for performing similarity query processing on the non-semantic index according to the non-semantic features of the image to be queried to obtain a similar non-semantic index, and performing similarity query processing on the semantic index according to the semantic features of the image to be queried to obtain a similar semantic index; The similarity query module is further configured to use the sample images corresponding to the similar non-semantic indexes and at least part of the sample images corresponding to the similar semantic indexes as similar sample images of the image to be queried.

14. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, used to implement the artificial intelligence-based image processing method described in any one of claims 1 to 12 when executing the executable instructions stored in the memory.

15. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the artificial intelligence-based image processing method described in any one of claims 1 to 12 when executed by a processor.

Citation Information

Patent Citations

  • Video semantic gateway

    CN102402612A

  • Image retrieval method based on object detection

    CN107256262A

  • Model generation method, video search method, device, terminal, and storage medium

    CN109635157A

  • Video retrieval method, device and equipment and storage medium

    CN111639228A

  • Image retrieval method, device, equipment and readable storage medium

    CN111753116A