Image processing methods, apparatus, electronic devices, computer-readable storage medium, and computer program product

Abstract

Image processing methods, an apparatus, electronic devices, a computer-readable storage medium and a computer program product, which are applied to various processing scenarios, such as cloud technology, artificial intelligence, intelligent traffic, assisted driving, in-vehicle devices, and maps, etc. An image processing method comprises: acquiring training data of a model to be trained, the training data comprising a sample text, a sample image and sample tags, the sample tags comprising detection box tags of the sample image; clustering the detection box tags to obtain N initial anchor box dimensions; using said model to execute the following processing: acquiring a text sample feature of the sample text, and performing detection on the sample image in combination with the N initial anchor box dimensions and the text sample feature, so as to obtain a sample detection result; and, on the basis of the differences between the sample detection result and the sample tags, training said model to obtain an image processing model. The present application can improve the model training efficiency.

Description

Image processing method, device, electronic device, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202311225352.2 and application date of September 21, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to image processing technology in the field of computer vision, and in particular to an image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0004] When training artificial neural network models for image processing tasks, random target queries are often used to process sample images in the training data. That is, sample images are detected based on random anchor box sizes. This affects the convergence speed of the artificial neural network model and thus the model training efficiency.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve model training efficiency.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] An embodiment of the present application provides an image processing method, which is performed by a first electronic device and includes:

[0009] Obtaining training data for a model to be trained, wherein the model to be trained is an artificial neural network model to be trained, the model to be trained is used for image processing, the training data includes sample text, sample images, and sample labels, and the sample labels include detection box labels of the sample images;

[0010] Clustering the detection box labels to obtain N initial anchor box sizes, where N is a positive integer;

[0011] The following processing is performed using the model to be trained: obtaining text sample features of the sample text, detecting the sample image in combination with the N initial anchor box sizes and the text sample features, and obtaining a sample detection result;

[0012] Based on the difference between the sample detection result and the sample label, the model to be trained is trained to obtain an image processing model, wherein the image processing model is used to obtain an image detection result based on the image to be detected and the image text prompt.

[0013] An embodiment of the present application further provides an image processing method, which is performed by a second electronic device and includes:

[0014] In response to an image processing request, obtaining an image to be processed and an image text prompt;

[0015] The image to be processed and the image text prompt are detected using an image processing model to obtain an image detection result, wherein the image processing model is trained using the image processing method provided in an embodiment of the present application.

[0016] An embodiment of the present application provides a first image processing device, comprising:

[0017] a data acquisition module configured to obtain training data of a to-be-trained model, wherein the to-be-trained model is an artificial neural network model to be trained, the to-be-trained model is used for image processing, the training data includes sample text, sample images, and sample labels, and the sample labels include detection box labels of the sample images;

[0018] a label clustering module configured to cluster the detection box labels to obtain N initial anchor box sizes, where N is a positive integer;

[0019] An image prediction module is configured to perform the following processing using the model to be trained: obtaining text sample features of the sample text, detecting the sample image in combination with the N initial anchor box sizes and the text sample features, and obtaining a sample detection result;

[0020] The model training module is configured to train the model to be trained based on the difference between the sample detection result and the sample label to obtain an image processing model, wherein the image processing model is used to obtain the image detection result based on the image to be detected and the image text prompt.

[0021] This embodiment of the present application provides a second image processing device, the second image processing device comprising:

[0022] A request response module configured to obtain an image to be processed and an image text prompt in response to an image processing request;

[0023] The image detection module is configured to use an image processing model to detect the image to be processed and the image text prompt to obtain an image detection result, wherein the image processing model is trained using the image processing method provided in the embodiment of the present application.

[0024] An embodiment of the present application provides a first electronic device for image processing, the first electronic device including:

[0025] A first memory, configured to store computer executable instructions or computer programs;

[0026] The first processor is configured to implement the image processing method applied to the first electronic device provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the first memory.

[0027] An embodiment of the present application provides a second electronic device for image processing, the second electronic device including:

[0028] a second memory for storing computer executable instructions or computer programs;

[0029] The second processor is configured to implement the image processing method applied to the second electronic device provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the second memory.

[0030] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a first processor, the computer-executable instructions or the computer program implement the image processing method provided in the embodiment of the present application for a first electronic device; or, when the computer-executable instructions or the computer program are executed by a second processor, the computer-executable instructions or the computer program implement the image processing method provided in the embodiment of the present application for a second electronic device.

[0031] An embodiment of the present application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or computer program are executed by a first processor, the image processing method provided in the embodiment of the present application and applied to a first electronic device is implemented; or, when the computer-executable instructions or computer program are executed by a second processor, the image processing method provided in the embodiment of the present application and applied to a second electronic device is implemented.

[0032] The embodiments of the present application have at least the following beneficial effects: when training a model to be trained for performing image processing tasks, N initial anchor frame sizes are determined by clustering results of detection frame labels, and sample images are detected based on the N initial anchor frame sizes to achieve training of the model to be trained; since the N initial anchor frame sizes are obtained from the detection frame labels during the above-mentioned model training process, accurate control of the training direction is achieved, thereby accelerating the convergence speed of the model and improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of an exemplary decoding structure;

[0034] FIG2 is another exemplary schematic diagram of a decoding structure;

[0035] FIG3 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;

[0036] FIG4 is a schematic structural diagram of a server in FIG3 provided in an embodiment of the present application;

[0037] FIG5 is a schematic structural diagram of a terminal in FIG3 provided in an embodiment of the present application;

[0038] FIG6 is a flowchart of an image processing method according to an embodiment of the present application;

[0039] FIG7 is a second flow chart of the image processing method provided in an embodiment of the present application;

[0040] FIG8 is a third flow chart of the image processing method provided in an embodiment of the present application;

[0041] FIG9 is a diagram of an exemplary visual application architecture provided in an embodiment of the present application;

[0042] FIG10 is a schematic diagram of an exemplary decoding structure provided in an embodiment of the present application;

[0043] FIG11 is a schematic diagram of an exemplary visual application provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0045] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first\second\third" are used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with the specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0049] 1) Artificial neural network is a mathematical model that imitates the structure and function of biological neural networks. Exemplary structures of artificial neural networks in the embodiments of this application include graph convolutional networks (GCN, a neural network for processing graph-structured data), deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), neural state machines (NSM), and phase-functioned neural networks (PFNN). The models to be trained and the image processing models involved in the embodiments of this application are all artificial neural network models.

[0050] 2) Known Classes: refers to the categories to which the labeled targets in the training data belong; the categories corresponding to the third target score in the embodiment of the present application may include known categories.

[0051] 3) Unknown Classes refers to the categories of targets that exist in the test data and the data to be tested but do not appear in the training data; the categories corresponding to the third target score in the embodiment of the present application may include unknown categories.

[0052] 4) Object Detection refers to the process of locating each target in an image and determining the target's category (including known categories and unknown categories); the image processing in the embodiment of the present application includes object detection.

[0053] 5) Open Set Object Detection: This refers to object detection performed on an open set of test data. This allows for the determination of both the location and category of objects of known categories, as well as the location of objects of unknown categories. The present invention may include development set object detection during the training of the model to be trained.

[0054] 6) Feature Pyramid Network (FPN), which is used together with ResNet to form a backbone network to extract multi-scale features of images.

[0055] 7) Unknown Probability refers to the probability that the detected target belongs to an unknown category, also known as the score of the unknown category.

[0056] 8) Feature Map refers to the features obtained by convolving an image with a filter. The feature map can also be further convolved with the filter to obtain a new feature map; for example, the initial image features in the embodiment of the present application.

[0057] 9) Latent Space: refers to a feature space composed of network latent features (outputs of the network's intermediate layers), such as the feature spaces in each encoder and decoder in the embodiments of the present application.

[0058] 10) Latent Region: refers to the region in the latent feature space.

[0059] 11) Regional features refer to the deep features obtained after the information passes through the fully connected layer, which are used for target detection.

[0060] 12) Encoding vector refers to the feature vector obtained by reducing the dimension of regional features through multiple fully connected layers; for example, the encoding vector of 1024-dimensional regional features is reduced to 128-dimensional.

[0061] It should be noted that in the field of computer vision, for example, in digit classification, an artificial neural network model takes an image as input and produces 10 outputs, each representing the probability of one of the digit classes. Therefore, there is a problem of task versatility. To improve versatility, we can continue to add more tasks to the above artificial neural network model, such as adding new prediction types and new datasets. In this case, the architecture is usually expanded by adding additional output heads. For example, when performing image classification (ImageNet classification) and object detection (Coco detection), the output heads include confidence output heads for 1000 categories, detection boxes for 80 categories, and output heads for the corresponding confidence levels. In this way, the number of output heads increases with the increase in tasks and datasets, thus affecting model training efficiency. In addition, each model application always generates 1000 confidence types and 80 detection boxes, which affects image detection efficiency.

[0062] To improve model training and image detection efficiency, natural language text can be used to define tasks, replacing multi-head outputs. For example, there are the Knowledge Question Answering (VQA) task: "What's sitting on the sofa?", the Object Detection and Localization task: "Find all instances of dogs," the Image Captioning task: "What's happening in the image?", and the Image Classification task: "What type of object is this?". However, while multi-task vision systems using natural language text can perform computer vision tasks (such as Knowledge Question Answering, image captioning, image classification, and object detection and localization), they rely on pre-trained models for visual language, such as a pre-trained end-to-end object detector (DETR). However, the object queries used in DETR lack clear physical meaning, which hinders model training convergence and, in turn, reduces model training efficiency.

[0063] For example, refer to Figure 1, which is an exemplary decoding structure diagram; as shown in Figure 1, image features (Image Features) 1-1 are used as values ​​(V, also called value features) to input the cross-attention mechanism (Cross-Attention) module 1-2 of each layer (layer 1 is shown as an example), positional encodings (Positional Encodings) 1-3 and image features 1-1 are used as keys (K, also called key features) to input the cross-attention mechanism module 1-2 of each layer, and initialized decoder embeddings (Edcoder Embeddings) 1-4 and learnable object queries (Learnable Queries) 1-5 are used as query features (Q1) to input the cross-attention mechanism module 1-2 of each layer; and then the output of the cross-attention mechanism module 1-2 is combined to process the next layer (for example, layer 2).

[0064] In addition, the learnable object query can also be set as a random anchor box. For example, see Figure 2, which is another exemplary decoding structure diagram; as shown in Figure 2, the image feature 2-1 is used as a value input to the cross-attention mechanism module 2-2 of each layer (layer 1 is shown as an example), the position encoding 2-3 and the image feature 2-1 are used as keys to input the cross-attention mechanism module 2-2 of each layer, the initialized decoder embedding 2-4 and the learnable object query 2-5 (x (0) ,y (0) , h (0) , w (0) ) is used as the query feature input to the cross-attention mechanism module 2-2 of each layer; and then the output of the cross-attention mechanism module 2-2 is combined with the output of the next layer (for example, layer 2), for example, the offset (Δx, Δy, Δh, Δw) output by layer 1 is used to compare with (x (0) ,y (0) , h (0) , w (0) ) to obtain the object query (x (1) ,y (1) , h (1) , w (1) ). Among them, (x (0) ,y (0) , h (0) , w (0) ) in (x (0) ,y (0) ) is used as a position query (Position Query), (h (0) , w (0) ) is used to adjust the cross-attention mechanism module 2-2.

[0065] It should be noted that the Dynamic Anchor Box Detector (DAB-DETR) in Figure 2 proposes an object query using dynamic anchor boxes (DABs), which are updated layer by layer. This method leverages explicit position priors to improve the similarity between the object query and features, accelerating model convergence. Furthermore, the length and width of the anchor box are used in the attention map corresponding to the position, which is equivalent to using layer-by-layer soft region of interest (Soft ROI Pooling) to continuously adjust the object query. However, issues with model convergence efficiency still exist.

[0066] Based on this, the embodiments of the present application provide an image processing method, apparatus, equipment, computer-readable storage medium and computer program product, which can improve the efficiency of model training. The exemplary application of the equipment provided by the embodiments of the present application is described below. The first electronic device for image processing (hereinafter referred to as the model training device) and the second electronic device for image processing (hereinafter referred to as the model application device) provided by the embodiments of the present application can be implemented as various types of terminals such as smart phones, smart watches, laptops, tablet computers, desktop computers, smart home appliances, set-top boxes, smart car devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices and smart speakers. They can also be implemented as servers or a combination of the two. Below, the exemplary application when the model training device is implemented as a server and the model application device is implemented as a terminal will be described.

[0067] Referring to FIG3 , FIG3 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application. As shown in FIG3 , to support an image processing application, in an image processing system 100, a terminal 200 (terminal 200-1 and terminal 200-2 are shown as examples) is connected to a server 400 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. In addition, the image processing system 100 also includes a database 500 for providing data support to the server 400. FIG3 shows a case where the database 500 is independent of the server 400. In addition, the database 500 can also be integrated into the server 400, which is not limited in the embodiment of the present application.

[0068] Terminal 200 is used to obtain the image to be processed and the image text prompt in response to the image processing request; use the image processing model to detect the image to be processed and the image text prompt to obtain the image detection result, wherein the image processing model is sent by the server 400 through the network 300, and display the image to be processed, the image text prompt and the image detection result (graphic interface 200-11 and graphical interface 200-21 are shown as examples).

[0069] Server 400 is used to obtain training data for the model to be trained, where the training data includes sample text, sample images and sample labels, where the sample labels include detection frame labels of the sample images; cluster the detection frame labels to obtain N initial anchor frame sizes; use the model to be trained to perform the following processing: detect the sample images in combination with the N initial anchor frame sizes and text sample features of the sample text to obtain sample detection results; train the model to be trained based on the difference between the sample detection results and the sample labels to obtain an image processing model; and send the image processing model to the terminal 200 via the network 300.

[0070] In some embodiments, server 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0071] Referring to FIG. 4 , FIG. 4 is a schematic diagram of the structure of a server in FIG. 3 provided in an embodiment of the present application; as shown in FIG. 4 , the server 400 includes: at least one first processor 410, a first memory 450, and at least one first network interface 420. The various components in the server 400 are coupled together via a first bus system 440. It will be understood that the first bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG. 4 , all the various buses are labeled as the first bus system 440.

[0072] The first processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0073] The first memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The first memory 450 may optionally include one or more storage devices physically located away from the first processor 410.

[0074] The first memory 450 includes a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The first memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0075] In some embodiments, the first memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0076] The first operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and process hardware-based tasks;

[0077] A first network communication module 452 is configured to communicate with other electronic devices via one or more (wired or wireless) first network interfaces 420 . Exemplary first network interfaces 420 include Bluetooth, Wi-Fi, and USB.

[0078] In some embodiments, the first image processing device provided in the embodiments of the present application can be implemented in software. FIG4 shows a first image processing device 455 stored in a first memory 450. The first image processing device 455 can be software in the form of a program or plug-in, and includes the following software modules: a data acquisition module 4551, a label clustering module 4552, an image prediction module 4553, a model training module 4554, and a size acquisition module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0079] Referring to Figure 5 , Figure 5 is a schematic diagram of the structure of a terminal in Figure 3 provided in an embodiment of the present application; as shown in Figure 5 , the terminal 200 includes: at least one second processor 210, a second memory 250, at least one second network interface 220, and a user interface 230. The various components in the terminal 200 are coupled together via a second bus system 240. It will be understood that the second bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the second bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 5, all various buses are labeled as the second bus system 240.

[0080] The second processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0082] The second memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The second memory 250 may optionally include one or more storage devices physically located away from the second processor 210.

[0083] The second memory 250 includes a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory, and the volatile memory may be a random access memory. The second memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0084] In some embodiments, the second memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0085] The second operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and process hardware-based tasks;

[0086] A second network communication module 252, for reaching other electronic devices via one or more (wired or wireless) second network interfaces 220, exemplary second network interfaces 220 include: Bluetooth, Wi-Fi, and Universal Serial Bus;

[0087] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0088] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0089] In some embodiments, the second image processing device provided in the embodiments of the present application can be implemented using software. FIG3 shows a second image processing device 255 stored in the second memory 250. The second image processing device 255 can be software in the form of a program or plug-in, and includes the following software modules: a request response module 2551 and an image detection module 2552. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0090] In some embodiments, the first image processing device and the second image processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the first image processing device and the second image processing device provided in the embodiments of the present application can be processors in the form of hardware decoding processors, which are programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0091] In some embodiments, the terminal can implement the image processing method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, the computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as an image APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0092] Below, we will explain the image processing method provided by the embodiment of this application in conjunction with the exemplary application and implementation of the model training device and model application device provided in the embodiment of this application. In addition, the image processing method provided by the embodiment of this application is applied to various image processing scenarios such as cloud technology, artificial intelligence, smart transportation, vehicle-mounted and mapping.

[0093] Refer to Figure 6, which is a flow chart of the image processing method provided in an embodiment of the present application, wherein the execution body of each step is the model training device; the steps shown in Figure 6 will be explained below.

[0094] Step 101: Obtain training data for the model to be trained, wherein the training data includes sample text, sample images, and sample labels, and the sample labels include detection box labels of the sample images.

[0095] In an embodiment of the present application, a model training device obtains a data set for training a model to be trained, and the obtained data set for training the model to be trained is referred to as training data.

[0096] It should be noted that sample images are images in the training data that are to be processed, such as images to be classified, images to be described, images to be subjected to object detection, images to be subjected to question-and-answer (Q&A), etc. Sample text is the text prompt associated with the sample image, used to determine the processing direction and results of the image processing task. For example, in a knowledge Q&A scenario, sample text can be prompt text that asks a question about the image to be Q&A; in an image description scenario, sample text can be prompt text that instructs the image to be described; in an object detection and localization scenario, sample text can be prompt text that instructs the image to be detected to be subjected to object detection; and in an image classification scenario, sample text can be prompt text that instructs the image to be classified to be classified. Here, the sample images and sample text are combined to form the input data for the artificial neural network model. Sample labels are the annotated data of the sample images and sample text, representing actual image processing results, and include at least one of the following: a detection box label for the sample image, a text output result label, a category label, etc. In addition, the model to be trained is an artificial neural network model to be trained, and the model to be trained is used for image processing; here, the model to be trained can be the constructed original artificial neural network model, or it can be a pre-trained artificial neural network model, etc., and the embodiments of the present application are not limited to this.

[0097] It should also be noted that the image processing described in the embodiment of the present application refers to the processing of detecting images in combination with text to obtain detection content that matches the text description; wherein, image processing includes at least one of the following: knowledge question and answer, image description, target detection and positioning, and image classification.

[0098] Step 102: Cluster the detection box labels to obtain N initial anchor box sizes.

[0099] It should be noted that the model training device clusters the detection frame labels to cluster detection frames of the same size into one category; and determines N detection frame sizes based on the clustering results of each category as N initial anchor frame sizes.

[0100] In an embodiment of the present application, a model training device clusters detection box labels based on a size dimension to cluster multiple detection box size categories, and determines a detection box size based on each detection box size category to obtain M detection box sizes. N detection box sizes are selected from the M detection box sizes as N initial anchor box sizes, where N is a positive integer, M ≥ N, and M is an integer.

[0101] It should be noted that the initial anchor box size is the size of the preset query target area; and the query target is the entity existing in the preset sample image, such as animals, people, objects and scenes in the sample image.

[0102] In step 102 of an embodiment of the present application, the model training device clusters the detection frame labels to obtain N initial anchor frame sizes, including: the model training device clusters the detection frame labels based on the size dimension to obtain M types of clustering results (called M detection frame size categories); and obtains M detection frame sizes corresponding to the M types of clustering results; then, from the detection frame labels, counts the number of detection frames of each detection frame size; finally, from the M detection frame sizes, selects the N detection frame sizes with the largest number of detection frames; based on the N detection frame sizes, determines the N initial anchor frame sizes.

[0103] It should be noted that when the model training device clusters the detection frame labels, it does so based on the size dimension to cluster M clusters, which are M clustering results; wherein, in a clustering result, the sizes of the detection frames are similar, while the size difference between two detection frames belonging to different clustering results is greater than the difference threshold. Here, the model training device determines a detection frame size that represents the clustering result for the sizes corresponding to each detection frame in each clustering result; for determining the detection frame size, the model training device can determine it by selecting any one detection frame, or by finding the average size of each detection frame in the clustering result, etc., which is not limited in this embodiment of the present application; finally, the model training device can obtain M detection frame sizes for the M clustering results. In addition, the model training device can determine each of the M detection frame sizes as the initial anchor frame size, and can also determine some of the detection frame sizes selected from the M detection frame sizes as N initial anchor frame sizes, which is not limited in this embodiment of the present application. When the model training device determines some of the detection box sizes selected from the M detection box sizes as N initial anchor box sizes, the selection is based on the number of detection boxes, which refers to the number of detection boxes in the detection box label whose size is the detection box size.

[0104] In an embodiment of the present application, the model training device performs the following processing (step 103) using the model to be trained to obtain the result of image processing of the sample text and the sample image based on the model to be trained.

[0105] Step 103: Obtain text sample features of the sample text, and detect the sample image in combination with the N initial anchor box sizes and the text sample features to obtain a sample detection result.

[0106] In an embodiment of the present application, the model training device can extract features of the sample text through the model to be trained, and the extracted features are called text sample features; then, the model training device determines the target query frames corresponding to each position point based on the N initial anchor frame sizes, and finally obtains multiple target query frames for the sample image; then, the features for target detection are determined through the multiple target query frames, target detection is performed based on the determined features for target detection, and image processing is performed to match the content described by the sample text in combination with the determined features for target detection and the text sample features, thereby obtaining a sample detection result.

[0107] It should be noted that the sample detection result refers to the result of image processing of the sample text and sample image based on the model to be trained.

[0108] Refer to Figure 7, which is a second flow chart of the image processing method provided in an embodiment of the present application, wherein the executor of each step is a model training device; as shown in Figure 7, in an embodiment of the present application, step 103 can be implemented through steps 1031 to 1034; that is, the model training device obtains the text sample features of the sample text, combines N initial anchor frame sizes and text sample features, detects the sample image, and obtains a sample detection result, including steps 1031 to 1034. Each step is explained below.

[0109] Step 1031: Perform region coding on the initial image features of the sample image to obtain initial region features.

[0110] In an embodiment of the present application, the model training device can extract the features of the overall dimension of the sample image through the model to be trained, and the extracted features are called initial image features; then, the model training device performs region encoding on the initial image features to optimize the initial image features; region encoding is used to convert the initial image features into spatial features of the local region position dimension of the sample image; here, the region encoding result is called initial region feature.

[0111] It should be noted that the initial image features are spatial features of the overall dimension of the sample image, and are basic feature representations used for image processing of the sample image.

[0112] Step 1032: Determine Q target query boxes based on the N initial anchor box sizes and the P specified targets.

[0113] In an embodiment of the present application, the model training device can obtain P designated targets, each designated target corresponding to a pre-set target, and the plurality of designated targets is the target that is pre-set for the sample image and is included in the sample image at most. Here, the model training device determines a target query box of N initial anchor box sizes for each designated target. Ultimately, for the P designated targets, P*N target query boxes can be obtained. Therefore, Q = P*N.

[0114] It should be noted that each designated target is a preset position in the sample image. The model training device takes the designated target as the center and each initial anchor frame size as the size to determine the region frame. The determined region frame is the target query frame; thus, for N initial anchor frame sizes and P designated targets, P*N target query frames can be obtained.

[0115] In an embodiment of the present application, the target query box includes the following information: an anchor point and an anchor box, wherein the anchor point represents the location point of the specified target in the sample image, and the anchor box represents the size of the area box centered on the anchor point.

[0116] Step 1033: Perform attention processing on the initial region features, the initial image features, and the Q target query boxes to obtain target region features.

[0117] In an embodiment of the present application, the model training device can obtain the features of each target query box through initialization, and combine the target query box and the features of the target query box into a query feature. By comparing the similarity of the query feature with the initial region feature and the initial image feature, attention processing of the initial region feature, the initial image feature and Q target query boxes is realized to detect the target that corresponds to the target query box and is similar to the features of the target query box; and the feature used to characterize the target is the target region feature.

[0118] In an embodiment of the present application, the model training device combines the initial region features, the initial image features, and the Q target query frames for attention processing to obtain the target region features, including: the model training device combines the initial region features, the initial image features, and the Q target query frames to determine the key features, the value features, and the query features; performs attention processing on the key features, the value features, and the query features through the target decoder of the model to be trained to obtain Q query frame offsets; superimposes the Q query frame offsets on the Q target query frames to obtain Q target anchor frames; obtains the features corresponding to the Q target anchor frames to obtain the target region features. Among them, the target decoder is a decoder in the model to be trained for obtaining the target region features by combining the key features, the value features, and the query features, and the target decoder has one layer; of course, the target decoder can also have multiple layers.

[0119] In an embodiment of the present application, the key features, value features and query features are determined in combination with the initial region features, the initial image features and the Q target query boxes, including: the model training device determines the key features based on the initial region features and the initial image features; and determines the value features based on the initial image features; and determines the query features based on the Q target query boxes and the designated content features corresponding to each of the Q target query boxes.

[0120] It should be noted that the attention processing performed by the model training device on the initial region features, initial image features, and Q target query frames is a decoding process based on cross-attention; wherein, the initial region features and initial image features are used to determine the key features, the initial image features are used to determine the value features, and the features of the target query frame and the initialized target query frame (called the specified content features) are used to determine the query features, and then attention processing is performed on the value features, key features, and query features; here, the attention processing results corresponding to the initial region features, initial image features, and Q target query frames are Q query frame offsets, and the Q query frame offsets correspond one-to-one to the Q target query frames; thus, the model training device superimposes the Q query frame offsets on the Q target query frames one-to-one, thus completing the processing of superimposing the corresponding query frame offset on each target query frame; the superposition result of each target query frame and the corresponding query frame offset is a target anchor frame, thereby obtaining Q target anchor frames corresponding to the Q target query frames. In addition, the target region features include the features corresponding to each target anchor frame in the multiple target anchor frames.

[0121] Step 1034: Obtain text sample features of the sample text, perform image detection in combination with the target area features and the text sample features of the sample text, and obtain a sample detection result.

[0122] In an embodiment of the present application, the model training device can realize target detection of the sample image based on the target area features; then obtain the text sample features of the sample text, and combine the target detection results and the text sample features to obtain the image processing results corresponding to the sample image and sample text, which is the sample detection result.

[0123] In an embodiment of the present application, a model training device performs image detection in combination with target area features and text sample features to obtain a sample detection result, including: the model training device first predicts a detection frame based on the target area features to obtain a predicted detection frame; then performs attention processing on the target area features and the text sample features of the sample text to obtain associated features; and predicts a first target score of the predicted detection frame based on the target area features, predicts a second target score of the predicted detection frame based on the associated features, and combines the first target score and the second target score to obtain a third target score; and obtains a text prediction result in combination with the third target score and the associated features; finally, determines the sample detection result based on the text prediction result.

[0124] It should be noted that target region features refer to features used for target detection. Thus, the model training device, using the detection frame output head in the model to be trained, can predict the detection frame containing the target corresponding to the target region features, referred to as a predicted detection frame. Since the image processing task is defined by sample text, the model training device performs attention processing on the target region features based on the text sample features to extract features associated with the sample text from the target region features, namely, associated features. Furthermore, the model training device, using the target score output head in the model to be trained, predicts the category score of each predicted detection frame based on the target region features, thereby obtaining a first target score. This first target score is a score unrelated to the image processing task defined by the sample text. The model training device also uses the associated score output head in the model to train to predict the score of the predicted detection frame associated with the image processing task based on the associated features, thereby obtaining a second target score. This second target score is a score related to the image processing task defined by the sample text. The model training device then performs a weighted fusion of the first and second target scores to obtain a third target score, which is the final score related to the image processing task defined by the sample text. Since the third target score represents the score of the target to be processed in the image processing task, the model training device uses the text output head in the model to be trained to predict the third target score and associated features to obtain a text prediction result, which is used to describe the image processing result of the sample image.

[0125] It should also be noted that the sample detection results at least include text prediction results. In scenarios such as knowledge questions and answers, image description, and target detection and positioning, the sample detection results include text prediction results and sample images carrying detection frames, and the detection frames carried are the detection frames of the target objects corresponding to the sample text; that is, the model training device determines the sample detection results based on the text prediction results, including: the model training device determines the detection frame to be carried from the predicted detection frame based on the third target score, combines the detection frame to be carried with the sample image to obtain the sample image carrying the detection frame, and determines the text prediction results and the sample image carrying the detection frame as the sample detection results. In scenarios such as image classification, the sample detection results include text prediction results; that is, the model training device determines the sample detection results based on the text prediction results, including: the model training device determines the text prediction results as the sample detection results.

[0126] In an embodiment of the present application, the model training device performs attention processing on the target area features and the text sample features to obtain associated features, including: the model training device performs linear transformation on the target area features to obtain image linear features; and performs linear transformation on the text sample features to obtain text linear features; then, performs attention processing on the image linear features and the text linear features to obtain correlation weights; finally, superimposes the correlation weights and the image linear features to obtain associated features.

[0127] It should be noted that the model training device performs a linear transformation on the target region features and the text sample features, so that the linearly transformed target region features and the linearly transformed text samples are dimensionally consistent. The image linear features are the linearly transformed target region features, and the text linear features are the linearly transformed text samples. Here, the model training device performs attention processing on the image linear features and the text linear features to obtain features corresponding to the sample text from the image linear features, namely, the associated features.

[0128] In an embodiment of the present application, the model training device performs attention processing on the target area features and the text sample features to obtain associated features, including: the model training device pools the initial image features based on the predicted detection frame to obtain the target image features; and obtains the splicing features of the target image features and the target area features; finally, attention processing is performed on the splicing features and the text sample features of the sample text to obtain associated features.

[0129] It is understood that the model training device is also used to apply the predicted detection frame to the initial image features to improve the accuracy of the initial image features. In addition, because the target image features are features of the entire dimension of the sample image, by splicing the target image features and the target region features, the richness and comprehensiveness of the spliced ​​features can be improved. Image processing based on the spliced ​​features can then be performed to obtain sample detection results, which can improve the accuracy of image processing.

[0130] In step 103 of the embodiment of the present application, the model training device detects the sample image in combination with N initial anchor box sizes and text sample features to obtain a sample detection result, including: the model training device detects the sample image in combination with L specified anchor box sizes, N initial anchor box sizes and text sample features to obtain a sample detection result.

[0131] It should be noted that the model training device can obtain L specified anchor frame sizes that are different from the N initial anchor frame sizes after clustering the detection frame labels to obtain N initial anchor frame sizes, where L is a positive integer; thereby, the model training device can determine the target query frame by combining the L specified anchor frame sizes and the N initial anchor frame sizes.

[0132] It can be understood that the model training device improves the diversity of anchor box sizes by obtaining L specified anchor box sizes that are different from the N initial anchor box sizes, thereby improving the accuracy of image detection.

[0133] Step 104: Based on the difference between the sample detection results and the sample labels, the model to be trained is trained to obtain an image processing model.

[0134] In an embodiment of the present application, after obtaining the sample detection result, the model training device compares the sample detection result with the sample label to obtain the difference between the sample detection result and the sample label; since the difference between the sample detection result and the sample label represents the accuracy of the model to be trained, the model training device calculates the loss function value based on the difference between the sample detection result and the sample label, and performs backpropagation in the model to be trained based on the loss function value to adjust the model parameters in the model to be trained; in addition, the training of the model to be trained is iterative, and when the iterative training is completed, the model to be trained by the current iterative training is the image processing model. Among them, the image processing model is used to obtain image detection results based on the image to be detected and the image text prompt.

[0135] It should be noted that when the model training device determines that iterative training meets the training termination condition, it determines that iterative training is terminated; otherwise, iterative training continues. The training termination condition can be reaching an accuracy index threshold, reaching an iteration number threshold, reaching an iteration duration threshold, or a combination of the above, etc., and this embodiment of the application is not limited to this.

[0136] Refer to Figure 8, which is a flow chart of the image processing method provided in an embodiment of the present application. The execution subject of each step is the model application device. The steps shown in Figure 8 will be explained below.

[0137] Step 105: In response to the image processing request, obtain the image to be processed and the image text prompt.

[0138] In an embodiment of the present application, the model application device obtains the image processing model from the model training device and deploys the image processing model on its own device. Then, when the model application device receives an image processing request, it can use the deployed image processing model to perform image processing.

[0139] It should be noted that the image processing request is used to request that the image processing task indicated by the image text prompt be performed on the image to be processed; thus, the model application device can obtain the image to be processed and the image text prompt through the image processing request. Among them, the image to be processed is the image to be visually processed, and the image text prompt is used to describe the visual processing task of the image to be processed. In addition, in the knowledge question and answer scenario, the image text prompt can be a prompt text asking questions about the image to be processed, such as input text 11-21 to input text 11-24 in Figure 11; in the image description scenario, the image text prompt can be a prompt text indicating a description of the image to be processed, such as input text 11-25 to input text 11-28 in Figure 11; in the target detection and positioning scenario, the image text prompt can be a prompt text indicating target detection for the image to be processed, such as input text 11-29 to input text 11-212 in Figure 11; in the image classification scenario, the image text prompt can be a prompt text indicating classification of the image to be processed, such as input text 11-213 to input text 11-216 in Figure 11.

[0140] Step 106: Use the image processing model to detect the image to be processed and the image text prompt to obtain an image detection result.

[0141] It should be noted that the image detection result is the output result obtained by the model application device using the image processing model to perform image visual processing on the image to be processed and the image text prompt; it is easy to know that the image processing model is obtained by training the image training device through training data, and the image text prompt is the prompt text of any one of the following image processing tasks: knowledge question and answer, image description, target detection and positioning, and image classification.

[0142] In the embodiments of the present application, the model training device can be various servers, the model application device can be various servers or various terminals, and the model training device and the model application device can also be the same device, etc. The embodiments of the present application do not limit this.

[0143] The following describes an exemplary application of the present invention in a practical application scenario, which describes the process of determining anchor boxes based on clustering results of detection box labels to improve model training efficiency.

[0144] Referring to FIG9 , FIG9 is an exemplary visual application architecture diagram provided by an embodiment of the present application; as shown in FIG9 , it includes a visual encoder 9-11, a language encoder 9-12, a cross-modal encoder 9-13, a visual decoder 9-14 (i.e., an output head for boxes and scores), and a language decoder 9-15 (i.e., an output head for text, a Transformer decoder). Each module is described below.

[0145] The visual encoder 9-11 adopts the CNN backbone network 9-111, the target detector (Anchor-DETR) encoder (Anchor-DETR Encoder) 9-112 and decoder (Anchor-DETR Decoder) 9-113 and the region of interest (RoI) pooling 9-114.

[0146] Language encoder 9-12, encoded using pre-trained model 9-121 (BERT).

[0147] The cross-modal encoder 9-13 includes a linear layer 9-131, an attention module 9-132 (the attention module of the multimodal pre-trained model ViLBERT), a linear layer 9-133, and an association condition module 9-134. The attention module 9-132 can perform cross-contextualized representations of the visual encoder and the language encoder.

[0148] The visual decoder 9-14 includes a box output head (Box Head) 9-141, a target score output head (Objectness Head) 9-142 and a relevance score output head (Relatedness Head) 9-143 (called a relevance score output head).

[0149] Language decoder 9-15, including text decoder (Text Decoder) 9-151.

[0150] For the input image 9-2 (called the sample image), the backbone network 9-111 extracts convolutional features (called initial image features) and uses the extracted features as input to the encoder 9-112 and the region of interest pooling 9-114; wherein, the encoder 9-112 is used to process the input features to obtain context features (called initial region features) for each grid position, and use the context features and the object query 9-21 (called the target query box) as input to the decoder 9-113, thereby generating corresponding region descriptors (Region Descriptors) 9-22 (called target region features) for the object query 9-21 (R=100), and the region of interest pooling 9-114 processes the input features and the target box result 9-51 (called the predicted detection box) to obtain pooled features 9-23 (called target image features); then, the region descriptors 9-22 and the pooled features 9-23 are combined to obtain a complete domain encoding result 9-24 (called spliced ​​features). Here, the object query is used as learnable information, and non-maximum suppression (NMS) is eliminated in the encoder 9-112 and the decoder 9-113; and the region descriptor 9-22 includes position and limited appearance information. For the input text 9-3 (describing this image, called sample text), it is encoded by the pre-trained model 9-121 to obtain the encoded feature 9-31 (called the text sample feature). Then, the domain encoding result 9-24 is processed by the linear layer 9-131, and the encoded feature 9-31 is processed by the linear layer 9-133. The two processing results are used as input to the attention module 9-132 to obtain the cross-contextualized representation 9-41 (called the associated feature). Finally, the box output head 9-141 predicts the bounding box of the region descriptor 9-22 to obtain the target box result 9-51 (R candidate regions) for visual positioning and detection tasks; the target score output head 9-142 predicts the region descriptor 9-22 to obtain a task-independent score 9-52 (called the first target score); the relevance score output head 9-143 predicts the cross-contextualized representation 9-41 to obtain a task-related score 9-53 (called the second target score), and combines the score 9-53 and the score 9-52 to obtain the relevance score 9-54 (called the third target score); the relevance score 9-54 is also used to combine with the cross-contextualized representation 9-41 as the input of the association condition module 9-134 to output text 9-55 (a dog and a cat are lying on the bed, called the third target score).

[0151] It should be noted that the object detector models object detection as a set prediction problem and uses a bipartite matching label assignment strategy, enabling end-to-end (End 2 End) processing and reducing NMS post-processing. Anchor-DETR determines object queries based on the clustering results of detection boxes, which can improve the accuracy of candidate boxes. Its single-layer decoding process can accelerate model convergence and shorten training time.

[0152] See Figure 10, which is a schematic diagram of an exemplary decoding structure provided by an embodiment of the present application; as shown in Figure 10, the decoding structure 10-1 includes a layer of processing (layer 1). The image feature 10-11 (called the initial image feature) is used as a value input to the cross-attention mechanism module 10-12, the position encoding 10-13 (called the initial region feature) and the image feature 10-11 are used as keys to input the cross-attention mechanism module 10-12, the decoder embedding 10-14 (called the feature of the target query box) and the learnable object query 10-15 ((x (0) ,y (0) , h (0) , w (0) ), including the query object specified and the query object obtained by clustering the detection box label, called the target query box) is used as the query feature input to the cross-attention mechanism module 10-12. Here, the offset (Δx, Δy, Δh, Δw) output by layer 1 is used to compare with (x (0) ,y (0) , h (0) , w (0) ) to obtain a new object query (x (1) ,y (1) , h (1) , w (1) ). Among them, (x (0) ,y (0) ) is used as a position query (Position Query), (h (0) , w (0) ) is used to adjust the cross-attention mechanism modules 10-12.

[0153] It should be noted that by specifying the query object and using the query object obtained by clustering the detection box label as the object query, and setting the number of decoder layers to 1, the convergence speed of the model can be improved while ensuring the richness of the model in detecting multi-scale targets, thereby reducing the training cycle of the idol.

[0154] It should be noted that the backbone network of the model can be initialized using the pre-trained parameters of the image network (ImageNet), and the rest are initialized randomly. Here, when training the model, the stochastic gradient descent method (SGD) is used to solve the convolution template parameter w and the bias parameter b of the neural network model. In each iteration, the prediction result error is calculated and back-propagated to the convolutional neural network model, and the gradient is calculated and the parameters of the convolutional neural network model are updated. When in a training environment with 8 graphics processing units (GPUs), the learning rate of SGD can be set to 0.02, the batch size (Batch Size) can be set to 16 images, and two images per GPU. In addition, parallel training can be performed in a hardware environment (for example, GPU).

[0155] It should be noted that the architecture shown in Figure 9 can be applied to image-related content understanding services, such as knowledge question answering, image description, object detection and positioning, and image classification.

[0156] For example, refer to Figure 11, which is a schematic diagram of an exemplary visual application provided by an embodiment of the present application; as shown in Figure 11, in the knowledge question and answer application 11-11, for the input image and input text 11-21, the output result (called image detection result) 11-31 can be obtained; for the input image and input text 11-22, the output result 11-32 can be obtained; for the input image and input text 11-23, the output result 11-33 can be obtained; for the input image and input text 11-24, the output result 11-34 can be obtained.

[0157] In the image description application 11-12, for the input image and input text 11-25, an output result 11-35 can be obtained; for the input image and input text 11-26, an output result 11-36 can be obtained; for the input image and input text 11-27, an output result 11-37 can be obtained; for the input image and input text 11-28, an output result 11-38 can be obtained.

[0158] In the target detection and positioning application 11-13, for the input image and input text 11-29, the output result 11-39 can be obtained; for the input image and input text 11-210, the output result 11-310 can be obtained; for the input image and input text 11-211, the output result 11-311 can be obtained; for the input image and input text 11-212, the output result 11-312 can be obtained.

[0159] In the image classification application 11-14, for the input image 11-41 and the input text 11-213, the output result 11-313 can be obtained; for the input image 11-42 and the input text 11-214, the output result 11-314 can be obtained; for the input image 11-43 and the input text 11-215, the output result 11-315 can be obtained; for the input image 11-44 and the input text 11-216, the output result 11-316 can be obtained.

[0160] It is understandable that in the training process of the general vision model, determining the anchor box based on the clustering results of the detection box labels and reducing the number of decoding layers can speed up the convergence speed, reduce the training time, reduce the training cost, and improve the accuracy of detection and positioning tasks.

[0161] The following further describes an exemplary structure of the first image processing device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in FIG4 , the software module stored in the first image processing device 455 in the first memory 450 may include:

[0162] A data acquisition module 4551 is configured to obtain training data for a model to be trained, wherein the model to be trained is an artificial neural network model to be trained, the model to be trained is used for image processing, the training data includes sample text, sample images, and sample labels, and the sample labels include detection box labels of the sample images;

[0163] a label clustering module 4552 configured to cluster the detection box labels to obtain N initial anchor box sizes, where N is a positive integer;

[0164] The image prediction module 4553 is configured to perform the following processing using the model to be trained: obtaining text sample features of the sample text, detecting the sample image in combination with the N initial anchor box sizes and the text sample features, and obtaining a sample detection result;

[0165] The model training module 4554 is configured to train the model to be trained based on the difference between the sample detection result and the sample label to obtain an image processing model, wherein the image processing model is used to obtain the image detection result based on the image to be detected and the image text prompt.

[0166] In an embodiment of the present application, the label clustering module 4552 is further configured to cluster the detection frame labels based on the size dimension to obtain M types of clustering results, where M ≥ N and M is an integer; obtain M detection frame sizes corresponding to the M types of clustering results; count the number of detection frames of each detection frame size from the detection frame labels; select the N detection frame sizes with the largest number of detection frames from the M detection frame sizes; and determine the N initial anchor frame sizes based on the N detection frame sizes.

[0167] In this embodiment of the present application, the first image processing device 455 further includes a size acquisition module 4555 configured to acquire L specified anchor box sizes that are different from the N initial anchor box sizes, where L is a positive integer.

[0168] In an embodiment of the present application, the image prediction module 4553 is further configured to detect the sample image in combination with the L specified anchor frame sizes, the N initial anchor frame sizes and the text sample features to obtain the sample detection result.

[0169] In an embodiment of the present application, the image prediction module 4553 is further configured to perform region encoding on the initial image features of the sample image to obtain initial region features; determine Q target query frames in combination with the N initial anchor frame sizes and P specified targets, where P and Q are both positive integers; perform attention processing in combination with the initial region features, the initial image features and the Q target query frames to obtain target region features; obtain the text sample features of the sample text, and detect the sample image in combination with the target region features and the text sample features to obtain the sample detection result.

[0170] In an embodiment of the present application, the image prediction module 4553 is further configured to determine the key feature, the value feature and the query feature by combining the initial region feature, the initial image feature and the Q target query frames; perform attention processing on the key feature, the value feature and the query feature through the target decoder of the model to be trained to obtain Q query frame offsets; superimpose the Q query frame offsets on the Q target query frames to obtain Q target anchor frames; obtain the features corresponding to the Q target anchor frames respectively to obtain the target region feature.

[0171] In an embodiment of the present application, the image prediction module 4553 is further configured to determine the key feature based on the initial region feature and the initial image feature; determine the value feature based on the initial image feature; and determine the query feature based on the Q target query boxes and the designated content features corresponding to each of the Q target query boxes.

[0172] In an embodiment of the present application, the image prediction module 4553 is further configured to perform detection frame prediction based on the target area features to obtain a predicted detection frame; perform attention processing on the target area features and the text sample features to obtain associated features; predict a first target score of the predicted detection frame based on the target area features, predict a second target score of the predicted detection frame based on the associated features, and combine the first target score and the second target score to obtain a third target score; combine the third target score and the associated features to obtain a text prediction result; and determine the sample detection result based on the text prediction result.

[0173] In an embodiment of the present application, the image prediction module 4553 is further configured to perform a linear transformation on the target area features to obtain image linear features; perform a linear transformation on the text sample features to obtain text linear features; perform attention processing on the image linear features and the text linear features to obtain correlation weights; and superimpose the correlation weights and the image linear features to obtain the associated features.

[0174] In an embodiment of the present application, the image prediction module 4553 is further configured to pool the initial image features based on the predicted detection frame to obtain target image features; obtain the splicing features of the target image features and the target area features; and perform attention processing on the splicing features and the text sample features to obtain associated features.

[0175] In an embodiment of the present application, the target query box includes the following information: an anchor point and an anchor box, wherein the anchor point represents the location point of the designated target in the sample image, and the anchor box represents the size of the area box centered on the anchor point.

[0176] The following further describes an exemplary structure of the second image processing device 255 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in FIG5 , the software module stored in the second image processing device 255 in the second memory 250 may include:

[0177] The request response module 2551 is configured to obtain the image to be processed and the image text prompt in response to the image processing request;

[0178] The image detection module 2552 is configured to use an image processing model to detect the image to be processed and the image text prompt to obtain an image detection result, wherein the image processing model is trained using the image processing method provided in the embodiment of the present application.

[0179] In an embodiment of the present application, the image text prompt is a prompt text of any one of the following image processing tasks: knowledge question and answer, image description, target detection and positioning, and image classification.

[0180] An embodiment of the present application provides a computer program product, which includes computer-executable instructions or a computer program, which is stored in a computer-readable storage medium. A first processor of a first electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium, and the first processor executes the computer-executable instructions or computer program, so that the first electronic device performs the image processing method applied to the first electronic device described in the embodiment of the present application; or a second processor of a second electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium, and the second processor executes the computer-executable instructions or computer program, so that the second electronic device performs the image processing method applied to the second electronic device described in the embodiment of the present application.

[0181] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a first processor, the first processor will be caused to execute the application provided by the embodiment of the present application and the image processing method of the first electronic device; or, when the computer-executable instructions or the computer program are executed by a second processor, the second processor will be caused to execute the application provided by the embodiment of the present application and the image processing method of the second electronic device; for example, the image processing method shown in FIG. 6 .

[0182] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0183] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0184] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, e.g., in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0185] As an example, computer-executable instructions may be deployed to be executed on one electronic device (in this case, this one electronic device is the model training device and the model application device), or on multiple electronic devices located at one location (in this case, the multiple electronic devices located at one location are the model training device and the model application device), or on multiple electronic devices distributed at multiple locations and interconnected through a communication network (in this case, the multiple electronic devices distributed at multiple locations and interconnected through a communication network are the model training device and the model application device).

[0186] It is understandable that in the embodiments of this application, when data related to images and texts is involved, when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the information subject, and the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards. In addition, in this application, when the data capture technical solution involved is implemented, when the above embodiments of this application are applied to specific products or technologies, the collection, use and processing of relevant training data should comply with the requirements of laws and regulations, comply with the principles of legality, legitimacy and necessity, do not involve obtaining data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0187] To summarize, in the embodiment of the present application, when training a model to be trained for performing image processing tasks, N initial anchor frame sizes are determined by clustering results of detection frame labels, and sample images are detected based on the N initial anchor frame sizes to realize training of the model to be trained; since the N initial anchor frame sizes are obtained from the detection frame labels during the above-mentioned model training process, accurate control of the training direction is achieved, thereby accelerating the convergence speed of the model and improving the model training efficiency; in addition, during the model training process, the number of decoding layers is also reduced; thereby, the model training efficiency can be improved.

[0188] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method, the method being performed by a first electronic device, the method comprising: Acquire training data of a model to be trained, wherein the model to be trained is an artificial neural network model to be trained, the model to be trained is used for image processing, the training data includes sample text, sample images and sample labels, and the sample labels include detection box labels of the sample images; Clustering the detection box labels to obtain N initial anchor box sizes, where N is a positive integer; The model to be trained is used to perform the following processing: obtaining text sample features of the sample text, detecting the sample image in combination with the N initial anchor box sizes and the text sample features, and obtaining a sample detection result; Based on the difference between the sample detection result and the sample label, the model to be trained is trained to obtain an image processing model, wherein the image processing model is used to obtain an image detection result based on the image to be detected and the image text prompt.

2. The method according to claim 1, wherein: The clustering of the detection frame labels to obtain N initial anchor frame sizes includes: Clustering the detection box labels based on the size dimension to obtain M clustering results, where M ≥ N and M is an integer; Obtaining M detection box sizes corresponding to the M-class clustering results; Counting the number of detection frames of each detection frame size from the detection frame labels; Selecting N detection frame sizes having the largest number of detection frames from the M detection frame sizes; Based on the N detection box sizes, determine the N initial anchor box sizes.

3. The method according to claim 1 or 2, wherein: After clustering the detection frame labels to obtain N initial anchor frame sizes, the method further includes: Obtain L specified anchor box sizes that are different from the N initial anchor box sizes, where L is a positive integer; The combining the N initial anchor frame sizes and the text sample features to detect the sample image and obtain a sample detection result includes: The sample image is detected by combining the L specified anchor box sizes, the N initial anchor box sizes, and the text sample features to obtain the sample detection result.

4. The method according to claim 1 or 2, wherein: The acquiring of the text sample features of the sample text, and detecting the sample image in combination with the N initial anchor frame sizes and the text sample features to obtain a sample detection result, includes: Performing region coding on the initial image features of the sample image to obtain initial region features; Combining the N initial anchor box sizes and the P specified targets, determining Q target query boxes, where P and Q are both positive integers; Performing attention processing on the initial region feature, the initial image feature and the Q target query boxes to obtain a target region feature; The text sample feature of the sample text is obtained, and the sample image is detected in combination with the target area feature and the text sample feature to obtain the sample detection result.

5. The method according to claim 4, wherein: The combining the initial region feature, the initial image feature and the Q target query frames to perform attention processing to obtain the target region feature includes: Determine a key feature, a value feature, and a query feature by combining the initial region feature, the initial image feature, and the Q target query boxes; Performing attention processing on the key feature, the value feature, and the query feature through the target decoder of the model to be trained to obtain Q query box offsets; Superimposing Q query frame offsets on the Q target query frames to obtain Q target anchor frames; The features corresponding to the Q target anchor frames are obtained to obtain the target area features.

6. The method according to claim 5, wherein: The combining the initial region feature, the initial image feature and the Q target query boxes to determine the key feature, the value feature and the query feature includes: Determining the key feature based on the initial region feature and the initial image feature; determining the value feature based on the initial image feature; The query feature is determined based on the Q target query boxes and the designated content features corresponding to each of the Q target query boxes.

7. The method according to any one of claims 4 to 6, wherein: The detecting the sample image by combining the target area feature and the text sample feature to obtain the sample detection result includes: Perform detection frame prediction based on the target area features to obtain a predicted detection frame; Performing attention processing on the target area features and the text sample features to obtain associated features; Predicting a first target score of the predicted detection box based on the target region feature, predicting a second target score of the predicted detection box based on the associated feature, and combining the first target score and the second target score to obtain a third target score; Combining the third target score and the associated feature to obtain a text prediction result; Based on the text prediction result, the sample detection result is determined.

8. The method according to claim 7, wherein: The attention processing is performed on the target area feature and the text sample feature to obtain associated features, including: Performing linear transformation on the target area features to obtain image linear features; Performing linear transformation on the text sample features to obtain text linear features; Performing attention processing on the image linear features and the text linear features to obtain correlation weights; The correlation weight and the image linear feature are superimposed to obtain the association feature.

9. The method according to claim 7, wherein: The attention processing is performed on the target area feature and the text sample feature to obtain associated features, including: Pooling the initial image features based on the predicted detection frame to obtain target image features; Acquire a splicing feature of the target image feature and the target region feature; Attention processing is performed on the splicing features and the text sample features to obtain the associated features.

10. The method according to any one of claims 4 to 9, wherein: The target query frame includes the following information: an anchor point and an anchor frame, wherein the anchor point represents the position point of the designated target in the sample image, and the anchor frame represents the size of a region frame centered on the anchor point.

11. An image processing method, the method being performed by a second electronic device, the method comprising: In response to an image processing request, obtaining an image to be processed and an image text prompt; The image to be processed and the image text prompt are detected using an image processing model to obtain an image detection result, wherein the image processing model is trained using the image processing method according to any one of claims 1 to 10.

12. The method according to claim 11, wherein: The image text prompt is a prompt text of any one of the following image processing tasks: knowledge question and answer, image description, target detection and positioning, and image classification.

13. A first image processing device, the first image processing device comprising: A data acquisition module is configured to obtain training data of a model to be trained, wherein the model to be trained is an artificial neural network model to be trained, the model to be trained is used for image processing, the training data includes sample text, sample image and sample label, and the sample label includes a detection box label of the sample image; A label clustering module, configured to cluster the detection box labels to obtain N initial anchor box sizes, where N is a positive integer; The image prediction module is configured to perform the following processing using the model to be trained: obtaining text sample features of the sample text, detecting the sample image in combination with the N initial anchor box sizes and the text sample features, and obtaining a sample detection result; The model training module is configured to train the model to be trained based on the difference between the sample detection result and the sample label to obtain an image processing model, wherein the image processing model is used to obtain the image detection result based on the image to be detected and the image text prompt.

14. A second image processing device, the second image processing device comprising: A request response module, configured to obtain an image to be processed and an image text prompt in response to an image processing request; An image detection module is configured to detect the image to be processed and the image text prompt using an image processing model to obtain an image detection result, wherein the image processing model is trained using the image processing method described in any one of claims 1 to 10.

15. A first electronic device for image processing, the first electronic device comprising: A first memory, used to store computer executable instructions or computer programs; The first processor is configured to implement the image processing method according to any one of claims 1 to 10 when executing the computer executable instructions or computer program stored in the first memory.

16. A second electronic device for image processing, the second electronic device comprising: A second memory, used to store computer executable instructions or computer programs; The second processor is used to implement the image processing method described in any one of claims 11 and 12 when executing the computer executable instructions or computer programs stored in the second memory.

17. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein when the computer-executable instructions or the computer program are executed by a first processor, the image processing method described in any one of claims 1 to 10 is implemented; or, when the computer-executable instructions or the computer program are executed by a second processor, the image processing method described in any one of claims 11 and 12 is implemented.

18. A computer program product, comprising computer executable instructions or a computer program, wherein the computer executable instructions or the computer program are used to implement the image processing method described in any one of claims 1 to 10 when executed by a first processor; or, the computer executable instructions or the computer program are used to implement the image processing method described in any one of claims 11 and 12 when executed by a second processor.

Images