Image processing method, apparatus, device, storage medium and computer program product

Abstract

Embodiments of the present application relate to the field of artificial intelligence, and disclose an image processing method, device and equipment, a storage medium and a computer program product. The method comprises: performing visual feature extraction processing on a to-be-processed image to obtain a visual feature vector of the to-be-processed image; performing character prediction processing on the visual feature vector of the to-be-processed image to obtain N description character sequences; wherein each of the N description character sequences comprises M description characters, the first description character sequence in the N description character sequences is obtained according to the visual feature vector of the to-be-processed image, the (t+1)th description character sequence in the N description character sequences is obtained according to the first t description character sequences, M and N are integers greater than 1, and t is a positive integer less than or equal to N; and a predicted description character sequence of the to-be-processed image is generated according to the N description character sequences, so as to improve the generation rate of the predicted description character sequence used to describe the image content of the image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to an image processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Image captioning refers to the technique of describing the content of an image using natural language; that is, the technique of generating descriptive text (predicting descriptive character sequences) based on an image to describe its content. The accuracy and generation rate of the generated descriptive text are key concerns in the field of image captioning, but existing techniques for image captioning suffer from low generation rates. Summary of the Invention

[0003] This application provides an image processing method, apparatus, device, storage medium, and computer program product that can improve the generation rate of predictive descriptive character sequences used to describe image content.

[0004] On one hand, embodiments of this application provide an image processing method, including:

[0005] Visual feature extraction is performed on the image to be processed to obtain the visual feature vector of the image to be processed;

[0006] The visual feature vector of the image to be processed is subjected to character prediction processing to obtain N descriptive character sequences; wherein, each of the N descriptive character sequences includes M descriptive characters, the first descriptive character sequence of the N descriptive character sequences is obtained based on the visual feature vector of the image to be processed, the (t+1)th descriptive character sequence of the N descriptive character sequences is obtained based on the previous t descriptive character sequences, M and N are integers greater than 1, and t is a positive integer less than or equal to N;

[0007] Generate a predicted descriptive character sequence for the image to be processed based on the N descriptive character sequences.

[0008] On one hand, embodiments of this application provide an image processing apparatus, characterized in that the image processing apparatus includes an acquisition unit and a processing unit, the acquisition unit and the processing unit being used to execute the above-described image processing method.

[0009] On one hand, embodiments of this application provide an image processing device, characterized in that the image processing device includes an input interface and an output interface, and further includes:

[0010] A processor, adapted to implement one or more instructions; and,

[0011] A computer storage medium storing one or more instructions adapted to be loaded by the processor and executed by the image processing method described above.

[0012] On one hand, embodiments of this application provide a computer storage medium, characterized in that the computer storage medium stores computer program instructions, which, when executed by a processor, are used to perform the above-described image processing method.

[0013] On one hand, embodiments of this application provide a computer program product or a computer program, the computer program product including a computer program stored in a computer storage medium; the processor of the image processing device reads the computer program from the computer storage medium, the processor executes the computer program, causing the image processing device to perform the above-described image processing method.

[0014] In this embodiment, the image processing device can perform visual feature extraction processing on the image to be processed to obtain a visual feature vector of the image to be processed; then, perform character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences; and generate a predicted descriptive character sequence of the image to be processed based on the N descriptive character sequences. Each of the N descriptive character sequences includes M descriptive characters, the first descriptive character sequence is obtained from the visual feature vector of the image to be processed, the (t+1)th descriptive character sequence is obtained from the previous t descriptive character sequences, M and N are integers greater than 1, and t is a positive integer less than or equal to N. That is, the visual feature vector of the image to be processed can be processed multiple times to obtain multiple descriptive character sequences; then, a predicted descriptive character sequence for describing the image content of the image to be processed can be generated based on the obtained multiple descriptive character sequences. Each processing of the visual feature vector of the image to be processed yields a descriptive character sequence including multiple descriptive characters, which can reduce the number of processing steps for the visual feature vector of the image to be processed and improve the generation rate of the predicted descriptive character sequence for describing the image content. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of a feature region of an image to be processed according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of a predicted descriptive character sequence for generating an image to be processed, provided in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram of an image processing model provided in an embodiment of this application;

[0020] Figure 5 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0021] Figure 6 This is a schematic diagram illustrating the generation of a first descriptive character sequence based on an image to be processed, provided in an embodiment of this application.

[0022] Figure 7 This is a flowchart illustrating a training method for an initial image processing model provided in an embodiment of this application;

[0023] Figure 8 This is a schematic diagram illustrating the processing of label description character sequences of training sample images based on an initial image processing model, as provided in an embodiment of this application.

[0024] Figure 9 This is a schematic diagram of sequential masking using a multi-head self-attention layer with relaxed causal masking, provided in an embodiment of this application.

[0025] Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0026] Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0029] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0030] This application primarily relates to the field of Computer Vision (CV) within the field of artificial intelligence. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to transform the processed images into images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Image captioning is an important research area in computer vision. Image captioning refers to the technique of generating descriptive text (predictive descriptive character sequences) to describe the content of an image. To improve the generation rate of predictive descriptive character sequences for describing the content of an image, this application provides an image processing scheme. This scheme involves extracting visual features from the image to be processed to obtain a visual feature vector. Then, the visual feature vector is processed multiple times to obtain multiple descriptive character sequences. Finally, a predictive descriptive character sequence for describing the content of the image to be processed is generated based on these multiple descriptive character sequences. Each processing of the visual feature vector yields a descriptive character sequence containing multiple descriptive characters, which reduces the number of processing steps for the visual feature vector and thus improves the generation rate of the predictive descriptive character sequence for describing the content of the image.

[0031] In specific implementations, the image processing scheme provided in this application embodiment can be executed by an image processing device, which can be a terminal device or a server; the terminal device here may include, but is not limited to: computers, smartphones, tablets, laptops, vehicle terminals, etc.; the server here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0032] The image processing solution provided in this application has broad application prospects. For example, it can be used to quickly add text descriptions to images, or to help visually impaired people perceive the external world in real time. That is, the image processing solution provided in this application can convert an image into descriptive text that describes the image content. Then, it can be combined with speech synthesis technology to realize the conversion from descriptive text to speech, thereby helping visually impaired people perceive the image content.

[0033] Based on the above image processing scheme, this application provides an image processing method. See also... Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of this application. Figure 1 The image processing method shown can be executed by an image processing device. Figure 1 The image processing method shown may include the following steps:

[0034] S101, Perform visual feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed.

[0035] In one embodiment, the image to be processed can be an image containing arbitrary image content; the visual feature vector of the image to be processed is obtained by performing visual feature extraction processing on the image to be processed, and is a vector used to describe the features of the image content of the image to be processed; the visual feature vector of the image to be processed can indicate: the regional object features of the object included in the feature region of the image to be processed, and the context features of the object included in the feature region. The regional object features of the object included in the feature region of the image to be processed are used to describe the features of the object itself within the feature region, and the context features of the object included in the feature region are used to describe the features of the context environment in which the object is located within the feature region, that is, the relationship features between the object within the feature region and the image content of other regions in the image to be processed besides the feature region. The feature region of the image to be processed refers to the region containing the feature object identified by the image processing device. The feature region of the image to be processed can be a region of any shape containing the feature object, for example, a rectangular region, a circular region, etc. Figure 2 The diagram shown is a schematic diagram of a feature region of an image to be processed according to an embodiment of this application. The object "airplane" as indicated by 201 is a feature object in the image to be processed, and the feature region corresponding to the feature object can be as indicated by 202.

[0036] In one embodiment, the image processing device performs visual feature extraction processing on the image to be processed to obtain a visual feature vector of the image to be processed, which may include: performing regional object feature extraction processing on the image to be processed to obtain a regional object feature vector of the image to be processed; and performing context feature extraction processing on the regional object feature vector of the image to be processed to obtain a visual feature vector of the image to be processed.

[0037] S102, perform character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences.

[0038] In this context, each of the N descriptive character sequences includes M descriptive characters. The first descriptive character sequence in the N descriptive character sequences is obtained based on the visual feature vector of the image to be processed, and the (t+1)th descriptive character sequence in the N descriptive character sequences is obtained based on the previous t descriptive character sequences. M and N are integers greater than 1, and t is a positive integer less than or equal to N.

[0039] In one embodiment, the image processing device performs character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences. This may include: performing character prediction processing on the visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed to obtain a first descriptive character sequence; and performing character prediction processing on the visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first descriptive character sequence to obtain a second descriptive character sequence. That is, the image processing device can perform character prediction processing on the visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first t descriptive character sequences to obtain the (t+1)th descriptive character sequence, up to the Nth descriptive character sequence. The first reference descriptive character sequence includes K start characters, where K is a positive integer less than or equal to M; the M descriptive characters in the first descriptive character sequence are generated in parallel, the M descriptive characters in the second descriptive character sequence are generated in parallel, and the M descriptive characters in each of the N descriptive character sequences are generated in parallel. The start character is a character used to indicate the start of character prediction processing, specifically the character "". <start>The first reference description character sequence is a sequence of K start characters used to indicate the start of character prediction processing; for example, if K=2, the first reference description character sequence can be " <start> <start>It is worth noting that if the images to be processed are different, the number of descriptive character sequences obtained when performing character prediction processing on the visual feature vectors of different images to be processed may be different.

[0040] In specific implementation, when the image processing device performs character prediction processing on the visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first descriptive character sequence to obtain the second descriptive character sequence, it can specifically be obtained by performing character prediction processing on the visual feature vector of the image to be processed and the sequence composed of the first reference descriptive character sequence and the first descriptive character sequence. For example, assuming M = K = 2, that is, the first reference descriptive character sequence includes 2 start characters, and each descriptive character sequence includes 2 descriptive characters, that is, based on each processing of the visual feature vector of the image to be processed, a descriptive character sequence composed of 2 descriptive characters is obtained; such as Figure 3 The diagram illustrates a method for generating a predicted descriptive character sequence for an image to be processed, as provided in an embodiment of this application. The image processing device performs visual feature extraction processing on the image to be processed, as indicated by label 301, to obtain a visual feature vector of the image to be processed. Then, based on the visual feature vector of the image to be processed and a first reference descriptive character sequence ("...") corresponding to the image to be processed, including two start characters... <start> <start>The character prediction process is performed to obtain the first descriptive character sequence ("Y1 Y2"); then the visual feature vector of the image to be processed, and the sequence composed of the first reference descriptive character sequence and the first descriptive character sequence ("Y1 Y2") are obtained. <start> <start>The image processing device performs character prediction processing on the image to be processed ("Y1 Y2") to obtain the second descriptive character sequence ("Y3 Y4"), and so on until the Nth descriptive character sequence is obtained. Specifically, the image processing device predicts four descriptive character sequences based on the image to be processed as shown in 301. The first descriptive character sequence ("Y1 Y2") is "a airplane", the second descriptive character sequence ("Y3 Y4") is "that is", the third descriptive character sequence is "flying in", and the fourth descriptive character sequence is "the sky". Further, the predicted descriptive character sequence of the image to be processed generated based on these four descriptive character sequences is "a airplane that is flying in the sky".

[0041] S103, Generate a predicted descriptive character sequence for the image to be processed based on N descriptive character sequences.

[0042] In one embodiment, the predicted descriptive character sequence of the image to be processed is a text sequence used to describe the image content of the image to be processed. In a specific implementation, the image processing device can concatenate N descriptive character sequences to obtain the predicted descriptive character sequence of the image to be processed.

[0043] In this embodiment, the image processing device can perform visual feature extraction processing on the image to be processed to obtain a visual feature vector of the image to be processed; then, perform character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences; and generate a predicted descriptive character sequence of the image to be processed based on the N descriptive character sequences. Each of the N descriptive character sequences includes M descriptive characters, the first descriptive character sequence is obtained from the visual feature vector of the image to be processed, the (t+1)th descriptive character sequence is obtained from the previous t descriptive character sequences, M and N are integers greater than 1, and t is a positive integer less than or equal to N. That is, the visual feature vector of the image to be processed can be processed multiple times to obtain multiple descriptive character sequences; then, a predicted descriptive character sequence for describing the image content of the image to be processed can be generated based on the obtained multiple descriptive character sequences. Each processing of the visual feature vector of the image to be processed yields a descriptive character sequence including multiple descriptive characters, which can reduce the number of processing steps for the visual feature vector of the image to be processed and improve the generation rate of the predicted descriptive character sequence for describing the image content.

[0044] Based on the above image processing schemes and image processing method embodiments, this application provides another image processing method. The image processing method provided in this application can be implemented based on an image processing model; this image processing model can be built based on a machine translation (Transformer) model, or it can be built based on a convolutional neural network (CNN) with decoding capabilities; the image processing model includes an encoding module capable of performing visual feature extraction processing on the image, and a decoding module including character prediction processing on the visual feature vector of the image obtained through visual feature extraction processing by the encoding module. For example... Figure 4 The diagram shown is a schematic of an image processing model provided in an embodiment of this application. The image processing model is built based on the Transformer model. The encoding module of the image processing model may include a region-based fast convolutional neural network (Faster-RCNN) and L encoders consisting of multi-head self-attention layers and feed-forward layers. The decoding module of the image processing model may include an embedding layer, L decoders consisting of multi-head self-attention layers, multi-head cross-attention layers and feed-forward layers, a linear layer, and a normalization layer. Furthermore, the embedding layer is mainly used for output embedding and position embedding; the multi-head self-attention layer in the decoder can be a relaxed causal masked multi-head self-attention layer; the normalization layer can be a normalization layer using the softmax activation function, i.e., a softmax layer; L is a positive integer, for example, L can be 6, meaning the image processing model consists of 6 encoders and 6 decoders. See also Figure 5 This is a flowchart illustrating an image processing method provided in an embodiment of this application. Figure 5 The image processing method shown can be executed by an image processing device. Figure 5 The image processing method shown may include the following steps:

[0045] S501 uses a region-based fast convolutional neural network in the image processing model to extract regional object features from the image to be processed, thereby obtaining the regional object feature vector of the image to be processed.

[0046] S502 uses the encoder in the image processing model to perform contextual feature extraction on the feature vectors of regional objects in the image to be processed, thereby obtaining the visual feature vectors of the image to be processed.

[0047] In steps S501 to S502, the visual feature vector of the image to be processed can indicate: the regional object features of the objects included in the feature region of the image to be processed, and the contextual features of the objects included in the feature region. Specifically, the regional object features of the objects included in the feature region of the image to be processed describe the features inherent to the objects themselves within the feature region, while the contextual features of the objects included in the feature region describe the features of the context in which the objects within the feature region exist, i.e., the relationship between the objects within the feature region and the image content of other regions in the image to be processed besides the feature region.

[0048] In specific implementation, the image processing device uses encoders in the image processing model to perform context feature extraction on the regional object feature vectors of the image to be processed, and obtains the visual feature vector of the image to be processed. This can be done through L encoders. That is, the first encoder of the L encoders performs context feature extraction on the regional object feature vectors of the image to be processed, and then the second encoder performs context feature extraction on the output of the first encoder, and so on, until the Lth encoder performs context feature extraction on the output of the (L-1)th encoder, thereby obtaining the visual feature vector of the image to be processed. When processing through each of the L encoders, the processing is performed sequentially through the multi-head self-attention layer and the forward fully connected layer in the encoder.

[0049] S503, perform character prediction processing on the visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed to obtain the first descriptive character sequence.

[0050] The first reference descriptive character sequence includes K start characters; the first descriptive character sequence includes M descriptive characters, which are obtained based on the visual feature vector of the image to be processed. The M descriptive characters in the first descriptive character sequence are generated in parallel; K is a positive integer less than or equal to M.

[0051] In one embodiment, the image processing device performs character prediction processing on the visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed to obtain a first descriptive character sequence. This may include: performing feature interaction processing on the visual feature vector of the image to be processed and the first reference descriptive character sequence to obtain a prediction vector corresponding to the first reference descriptive character sequence; performing character probability prediction processing on the prediction vector corresponding to the first reference descriptive character sequence to obtain the probabilities of M groups of predicted characters; the probability of each predicted character in the M groups of predicted characters includes the probability corresponding to each predicted character within each group of predicted characters; based on the probabilities of the M groups of predicted characters, determining a descriptive character from each group of predicted characters to obtain a first descriptive character sequence. It should be noted that the prediction vector corresponding to the first reference descriptive character sequence is mainly used to indicate that the prediction vector at this time is obtained from the first reference descriptive character, and the finally determined descriptive characters are all used to describe the image to be processed; the subsequent step also includes obtaining the prediction vector corresponding to the first descriptive character sequence from the first descriptive character sequence, and then obtaining a second descriptive character sequence.

[0052] In a specific implementation, the image processing device performs feature interaction processing on the visual feature vector of the image to be processed and the first reference description character sequence to obtain the prediction vector corresponding to the first reference description character sequence. This may include: performing feature embedding processing on the first reference description character sequence through the embedding layer in the image processing model to obtain the embedding vector corresponding to the first reference description character sequence; and processing the visual feature vector of the image to be processed and the embedding vector corresponding to the first reference description character sequence through L decoders in the image processing model to obtain the prediction vector corresponding to the first reference description character sequence. In specific implementation, the image processing device processes the visual feature vector of the image to be processed and the embedding vector corresponding to the first reference description character sequence through the first decoder among L decoders. Then, it processes the visual feature vector of the image to be processed and the output of the first decoder through the second decoder, and so on, until the visual feature vector of the image to be processed and the output of the (L-1)th decoder are processed, thereby obtaining the prediction vector corresponding to the first reference description character sequence. During processing by each of the L decoders, the image is processed sequentially through a multi-head self-attention layer, a multi-head cross-domain attention layer, and a forward fully connected layer. The visual feature vector of the image to be processed is processed by inputting to the multi-head cross-domain attention layer in each decoder. Specifically, when the image processing device performs feature embedding processing on the first reference description character sequence through the embedding layer in the image processing model, it can perform output embedding processing and position embedding processing on the first reference description character sequence.

[0053] In one embodiment, if the image processing model is built based on a single decoder, the image processing device performs feature interaction processing on the visual feature vector of the image to be processed and the first reference description character sequence to obtain a prediction vector corresponding to the first reference description character sequence. This process may include: performing feature embedding processing on the first reference description character sequence through an embedding layer in the image processing model to obtain an embedding vector corresponding to the first reference description character sequence; performing feature masking processing on the embedding vector corresponding to the first reference description character sequence through a multi-head self-attention layer in the image processing model to obtain a masking vector corresponding to the first reference description character sequence; performing attention interaction processing on the visual feature vector of the image to be processed and the masking vector corresponding to the first reference description character sequence through a multi-head cross-domain attention layer in the image processing model to obtain an interaction vector corresponding to the first reference description character sequence; and performing fully connected processing on the interaction vector corresponding to the first reference description character sequence through a forward fully connected layer in the image processing model to obtain a prediction vector corresponding to the first reference description character sequence.

[0054] Furthermore, the image processing device performs character probability prediction processing on the prediction vector corresponding to the first reference description character sequence to obtain the probabilities of M groups of predicted characters. This can include: performing a linear transformation on the prediction vector corresponding to the first reference description character sequence through a linear layer in the image processing model; then inputting the output of the linear layer into a softmax layer for normalization processing to obtain the probabilities of the M groups of predicted characters; wherein, the probability of each predicted character in the M groups of predicted characters includes the probability corresponding to each predicted character within each group of predicted characters. Specifically, each predicted character within each group of predicted characters is determined from candidate characters, specifically the candidate characters whose probabilities are not zero among the candidate characters predicted by the image processing model; that is, the image processing device maps the output of the linear layer to the probabilities of M groups of candidate characters through the softmax layer in the image processing model, and the probability of each group of candidate characters includes the probabilities corresponding to all candidate characters; then, the candidate characters whose probabilities are not zero in each group of candidate characters are determined as predicted characters, thus obtaining the probabilities of the M groups of predicted characters. Among them, candidate characters can be characters included in dictionaries commonly used by professionals in fields such as image description or natural language processing (NLP).

[0055] Furthermore, the image processing device determines a descriptive character from each group of predicted characters based on the probabilities of the M groups of predicted characters to obtain the first descriptive character sequence. This may include: determining the predicted character indicated by the highest probability in each group of predicted characters as the descriptive character corresponding to each group of predicted characters to obtain the first descriptive character sequence.

[0056] For example, suppose M = K = 2, meaning the first reference descriptive character sequence includes 2 start characters, and each descriptive character sequence includes 2 descriptive characters. This means that each processing of the visual feature vector of the image to be processed yields a descriptive character sequence consisting of 2 descriptive characters. Figure 6 The diagram illustrates a method for generating a first descriptive character sequence from an image to be processed, as provided in this application embodiment. The image processing device uses a region-based fast convolutional neural network in the image processing model to perform region object feature extraction processing on the image to be processed, as indicated by label 601, to obtain a region object feature vector of the image to be processed. Through L encoders in the image processing model, context feature extraction processing is performed on the region object feature vector of the image to be processed to obtain a visual feature vector of the image to be processed. Finally, through the embedding layer in the image processing model, the first reference descriptive character sequence (" <start> <start>The image processing model performs feature embedding processing to obtain the embedding vector corresponding to the first reference description character sequence. Then, it uses L decoders to process the visual feature vector of the image to be processed and the embedding vector corresponding to the first reference description character sequence, obtaining the prediction vector corresponding to the first reference description character sequence. Each decoder uses a multi-head cross-domain attention layer to interact with the visual feature vector of the image to be processed. Next, a linear layer in the image processing model performs a linear transformation on the prediction vector corresponding to the first reference description character sequence. The output of the linear layer is then input to a softmax layer for normalization, resulting in two sets of predicted character probabilities. The probability of the character with the highest probability in each set is then used to determine the predicted character. The predicted characters are determined as the description characters corresponding to each group of predicted characters, resulting in the first description character sequence. If the vector elements representing the probabilities of the first group of predicted characters based on vector representation correspond to {a, one}, and the probabilities of the first group of predicted characters based on vector representation are {60%, 40%}, and the vector elements representing the probabilities of the second group of predicted characters based on vector representation correspond to {airplane, bird}, and the probabilities of the second group of predicted characters based on vector representation are {80%, 20%}, then the description character determined from the first group of predicted characters is "a", the description character determined from the second group of predicted characters is "airplane", and the resulting first description character sequence is "aairplane".

[0057] S504, perform character prediction processing on the visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first descriptive character sequence to obtain the second descriptive character sequence, and so on until the Nth descriptive character sequence is obtained.

[0058] In this case, the M descriptive characters included in the second descriptive character sequence are generated in parallel, and the M descriptive characters included in each descriptive character sequence are generated in parallel.

[0059] In one embodiment, when the image processing device performs character prediction processing on the visual feature vector of the image to be processed, the first reference description character sequence, and the first description character sequence to obtain the second description character sequence, it can specifically be obtained by performing character prediction processing on the visual feature vector of the image to be processed and the sequence composed of the first reference description character sequence and the first description character sequence; that is, the sequence composed of the first reference description character sequence and the first description character sequence is input into the decoding module of the image processing model; this process is similar to the above step S503, and will not be described again here.

[0060] S505, Generate a predicted descriptive character sequence for the image to be processed based on N descriptive character sequences.

[0061] Step S505 is the same as step S203 above, and will not be repeated here.

[0062] In this embodiment, the image processing device performs visual feature extraction processing on the image to be processed using an image processing model to obtain a visual feature vector of the image to be processed; then, it performs character prediction processing on the visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed to obtain a first descriptive character sequence; it performs character prediction processing on the visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first descriptive character sequence to obtain a second descriptive character sequence, and so on until the Nth descriptive character sequence is obtained; and a predicted descriptive character sequence of the image to be processed is generated based on the N descriptive character sequences; wherein, the first reference descriptive character sequence includes K start characters, each descriptive character sequence includes M descriptive characters, the M descriptive characters included in each descriptive character sequence are generated in parallel, M and N are integers greater than 1, and K is a positive integer less than or equal to M. In other words, the visual feature vector of the image to be processed can be processed multiple times by an image processing model to obtain multiple descriptive character sequences. Then, a predicted descriptive character sequence for describing the image content of the image to be processed can be generated based on the obtained multiple descriptive character sequences. Each processing of the visual feature vector of the image to be processed yields a descriptive character sequence that includes multiple descriptive characters. The multiple descriptive characters included in the descriptive character sequence are generated in parallel, which can reduce the number of processing times of the visual feature vector of the image to be processed and improve the generation rate of the predicted descriptive character sequence for describing the image content.

[0063] The image processing model provided in the above-described image processing method embodiments is obtained by training an initial image processing model. The initial image processing model has the same model structure as the aforementioned image processing model, but the model parameters are different. Based on this, this application embodiment provides a training method for the initial image processing model, see [link to relevant documentation]. Figure 7 This is a flowchart illustrating a training method for an initial image processing model provided in an embodiment of this application. Figure 7 The training method for the initial image processing model shown can be executed by an image processing device or by any electronic device capable of training the initial image processing model. In this application, the embodiment is described using an image processing device. Figure 7 The training method for the initial image processing model shown may include the following steps:

[0064] S701, Obtain training samples.

[0065] The training samples include: training sample images and target descriptive character sequences for the training sample images. The training sample images can be images different from the images to be processed, containing arbitrary image content, used to train the initial image processing model; the target descriptive character sequences for the training sample images are descriptive text used to describe the image content of the training sample images. Optionally, the training samples can be obtained from commonly used datasets in the field of image description, such as the commonly used MSCOCO dataset.

[0066] S702, taking the K start characters included in the label description character sequence of the training sample image as the basis, and with M as the increment, the label description character sequence of the training sample image is truncated sequentially to obtain P training description character sequences corresponding to the training sample image.

[0067] The label description character sequence of the training sample image consists of K start characters and the target description character sequence of the training sample image, where P is an integer greater than 1. Each of the P training description character sequences corresponding to the training sample image includes K start characters. For example, if the target description character sequence of a training sample image is "dog laying on ground" and K = 2, then the label description character sequence of the training sample image is "dog laying on ground". <start> <start>"dog laying on ground". It is worth noting that when the training sample images are different, the sequence length of the label description character sequence of different training sample images, which is composed of the target description character sequence of different training sample images, may be different; when K is a fixed value, when the label description character sequence of different training sample images with different sequence lengths is truncated, the number of training description character sequences corresponding to different training sample images will be different.

[0068] For example, assuming M = K = 2, if the target description character sequence of a training sample image is "doglaying on ground", and the label description character sequence of the training sample image is " <start> <start>"dog laying on ground"; The image processing device uses the two starting characters in the label description character sequence of the training sample image as a base, and increments by 2 to sequentially truncate the label description character sequence of the training sample image, resulting in three training description character sequences corresponding to the training sample image, namely " <start> <start> ”、" <start> <start>doglaying and <start> <start>dog laying on the ground.

[0069] S703, the training sample image is used as the input of the initial image processing model, and P training description character sequences are used as the input of the initial image processing model in sequence. The target description character sequence of the training sample image is used as the expected output to train the initial image processing model in order to optimize and obtain the image processing model.

[0070] In one embodiment, the image processing device uses training sample images as input to an initial image processing model, and sequentially uses P training descriptive character sequences as input to the initial image processing model, and the target descriptive character sequence of the training sample images as the expected output, to train the initial image processing model to optimize it. When optimizing the image processing model, the device uses training sample images as input to perform visual feature extraction on the training sample images to obtain visual feature vectors. Then, it sequentially uses P training descriptive character sequences as input to the initial image processing model, and sequentially performs feature interaction processing between each of the P training descriptive character sequences and the visual feature vector of the training sample images to obtain prediction vectors corresponding to each training descriptive character sequence. It then sequentially performs character probability prediction processing on the prediction vectors corresponding to each training descriptive character sequence to obtain the probabilities of M training predicted characters corresponding to each training descriptive character sequence. Based on the probabilities of the M training predicted characters corresponding to each training descriptive character sequence, the target descriptive character sequence of the training sample images, and the loss function, it trains the initial image processing model to optimize it to obtain the final image processing model. In this sequence, the sequence consisting of M training prediction characters corresponding to a training description character sequence is identical to the remaining training description character parts in the next training description character sequence of that training description character sequence, excluding the training description character sequence part. For example... Figure 8 The diagram illustrates a method for processing the label description character sequence of a training sample image based on an initial image processing model, as provided in this application embodiment. The training sample image is shown as labeled 801, the target description character sequence of the training sample image is "doglaying on ground," and the label description character sequence of the training sample image is "". <start> <start>The training description character sequence, obtained by truncating the label description character sequence of the training sample image according to a step size of M (i.e., increment), is input into the decoding module of the initial image processing model to obtain the probabilities of multiple training prediction characters, that is, the probability of each training prediction character in the sequence "dog laying on ground."; where "." represents the end character.

[0071] In practice, the image processing device uses the training sample image as input to the initial image processing model. The initial image processing model performs visual feature extraction on the training sample image to obtain the visual feature vector of the training sample image. This process is similar to the above process of performing visual feature extraction on the image to be processed through the image processing model to obtain the visual feature vector of the image to be processed, and will not be described again here.

[0072] Furthermore, taking the first training description character sequence out of P training description character sequences as an example, the first training description character sequence is used as the input to the initial image processing model. This first training description character sequence is then processed in conjunction with the visual feature vector of the training sample image to obtain the prediction vector corresponding to the first training description character sequence. Specifically, the image processing device can perform feature embedding processing on the first training description character sequence through the embedding layer in the initial image processing model to obtain the embedding vector corresponding to the first training description character sequence. Then, through the L decoders in the initial image processing model, the visual feature vector of the training sample image and the embedding vector corresponding to the first training description character sequence are processed to obtain the prediction vector corresponding to the first training description character sequence. Specifically, the image processing device can process the visual feature vector of the training sample image and the embedding vector corresponding to the first training description character sequence through the first decoder of the L decoders in the initial image processing model, then process the visual feature vector of the training sample image and the output of the first decoder through the second decoder, and so on, until the Lth decoder processes the visual feature vector of the training sample image and the output of the (L-1)th decoder, thereby obtaining the prediction vector corresponding to the first training description character sequence.

[0073] Furthermore, when the image processing device processes data through each of the L decoders in the initial image processing model, it sequentially passes through the multi-head self-attention layer, the multi-head cross-domain attention layer, and the forward fully connected layer in the decoder; the visual feature vectors of the training sample images are processed by inputting them into the multi-head cross-domain attention layer in each decoder. Even further, when the multi-head self-attention layer included in the decoder of the initial image processing model is a relaxed causal masking multi-head self-attention layer, processing can be performed sequentially through each of the L decoders in the initial image processing model, passing through the relaxed causal masking multi-head self-attention layer, the multi-head cross-domain attention layer, and the forward fully connected layer in the decoder. The initial image processing model employs a relaxed-constraint causal masking multi-head self-attention layer to construct the decoder. This is to perform sequential masking (i.e., sequence masking) on ​​the label description character sequences of the training sample images. This ensures that when processing one of the P training description character sequences, the initial image processing model only utilizes the features of that single training description character sequence from the label description character sequence of the training sample image, ignoring the features of the remaining training description character sequences. Consequently, when obtaining the probability of each of the M training prediction characters corresponding to that single training description character sequence, the features of that single training description character sequence are used. Therefore, the trained image processing model can generate M description characters to form the (t+1)th description character sequence based solely on the features of the first reference description character sequence and the sequence composed of the first t description character sequences, thus accelerating the generation speed of the description character sequence.

[0074] For example, such as Figure 9 The diagram shown is a schematic of sequential masking using a multi-head self-attention layer with relaxed causal masking, according to an embodiment of this application. Assume M = K = 2, and the label description character sequence of the training sample images is " <start> <start>"dog laying on ground" Figure 9 As shown by label 901, the probabilities of multiple trained predicted characters obtained through the initial image processing model are the probabilities of each trained predicted character in the sequence "dog laying on ground..", where "." represents the end character. The sequence "dog laying on ground.." in... Figure 9 As indicated by mark 902; by Figure 9 It can be seen that the probabilities of the two predicted characters "dog laying" corresponding to the first training description character sequence are based on... <start> <start>"The probability of the two trained predicted characters "on ground" corresponding to the second training description character sequence is obtained from the features it possesses; the probability of the second training predicted character is based on..." <start> <start>The probability of the three training prediction characters corresponding to the three training character sequences is obtained from the features of "dog laying"; the probability of the two training prediction characters ".." is based on the features of "dog laying". <start> <start>This is derived from the characteristics of a dog laying on the ground.

[0075] Furthermore, the image processing device sequentially performs character probability prediction processing on the prediction vector corresponding to each training description character sequence. When obtaining the probabilities of the M training prediction characters corresponding to each training description character sequence, the sequence formed by the M training prediction characters corresponding to a training description character sequence is identical to the remaining training description character parts in the next training description character sequence of that training description character sequence, excluding the parts of that training description character sequence itself. For example, if the label description character sequence corresponding to the training sample image is " <start> <start>"dog laying on ground", and M=K=2; use the first training description character sequence " <start> <start>For example, the first training description character sequence corresponds to the sequence "dog laying" formed by two training prediction characters, and the second training description character sequence... <start> <start>In "dog laying", except <start> <start>The remaining "doglaying" characters are the same except for "". In the specific implementation, taking the first training description character sequence as an example, the image processing device maps the output of the linear layer to the probability of M groups of candidate characters through the softmax layer in the initial image processing model. The probability of each group of candidate characters includes the probability of each candidate character in all candidate characters. Then, a training prediction character corresponding to each group of candidate characters is determined from the probability of each group of candidate characters, and the probabilities of the M training prediction characters corresponding to the first training description character sequence are obtained.

[0076] Furthermore, the image processing device trains the initial image processing model based on the probabilities of the M training predicted characters corresponding to each training descriptive character sequence, the target descriptive character sequence of the training sample image, and the loss function. When optimizing the image processing model, the device can determine the value of the loss function based on the probabilities of the M training predicted characters corresponding to each training descriptive character sequence and the target descriptive character sequence of the training sample image, and train the initial image processing model in the direction of reducing the value of the loss function to optimize the image processing model. Wherein, when M = K, the training function can be given by Equation 1:

[0077]

[0078] Where K is the number of start characters in the label description character sequence of the training sample image; P is the number of training description character sequences obtained by truncating the label description character sequence of the training sample image. If the sequence length of the target description character sequence of the training sample image is T, then... t is a variable used to indicate the t-th group of training prediction characters (i.e., the M training prediction characters corresponding to the t-th training description character sequence), where the number of characters in a group of training prediction characters is M; y i The i-th character in the target description character sequence of the training sample image; I1 indicates the visual features of the training sample image; θ1 represents the model parameters used by the initial image processing model when processing this training sample image; G ＜t This represents the sequence of predicted characters from the first t-1 training sets. For example, if the target description character sequence of the training sample image is "dog laying on ground", and the label description character sequence of the training sample image is " <start> <start>The predicted sequence is "dog laying on ground.", K=2; therefore, T=4. P=3, the variable t ranges from 1 to 2, the first training prediction character is "doglaying", and the second training prediction character is "on ground".

[0079] In one embodiment, the image processing model can be obtained by training an initial image processing model, or by optimizing and adjusting an intermediate image processing model obtained from training the initial model. The intermediate image processing model has the same model structure as the initial image processing model, but the model parameters are different. Based on this, the image processing device optimizes and adjusts the intermediate image processing model to obtain the final image processing model. This optimization can include: acquiring an adjustment sample image; using the adjustment sample image as input to the intermediate image processing model to obtain H sets of character prediction results; wherein each set of character prediction results in the H sets includes: Q result description character sequences, each of the Q result description character sequences includes M result description characters, and the (z+1)th result description character sequence in the Q result description character sequences is obtained based on the previous z result description character sequences, where H and Q are integers greater than 1, and z is a positive integer less than or equal to Q; and based on the H sets of character prediction results and the gradient function, the intermediate image processing model is optimized and adjusted to obtain the final image processing model. The conditioning sample image can be an image containing arbitrary image content, different from the training sample image and the image to be processed, used to optimize the intermediate image processing model. Optionally, the conditioning sample image can be obtained from a commonly used dataset in the field of image description, such as the commonly used MSCOCO dataset, and can be an image different from the training sample image.

[0080] In one embodiment, the image processing device uses the adjusted sample image as input to an intermediate image processing model. Each of the H sets of character prediction results is a descriptive text predicted by the intermediate image processing model to describe the image content of the adjusted sample image. Each set of character prediction results is a character sequence, and each set of character prediction results in the H sets is different. It is worth noting that if the adjusted sample images are different, using different adjusted sample images as input to the intermediate image processing model may result in different numbers of descriptive character sequences included in each set of character prediction results in the corresponding H sets of character prediction results. Different values ​​of H can be set according to different optimization and adjustment requirements for the intermediate image processing model; for example, H can be set to 5. Specifically, the image processing device can input the adjusted sample image into the intermediate image processing model H times, so that the adjusted sample image is processed H times by the intermediate image processing model to obtain H sets of character prediction results.

[0081] The process of processing the adjusted sample image once using an intermediate image processing model to obtain a set of character prediction results is similar to the process of processing the image to be processed using an image processing model to obtain a predicted descriptive character sequence for the image to be processed. However, the difference lies in the following: when the image processing model processes the image to be processed and obtains any one of N descriptive character sequences, the image processing device determines the predicted character indicated by the highest probability in each of the M sets of predicted characters corresponding to that any one descriptive character sequence as the descriptive character corresponding to each set of predicted characters, thus obtaining that any one descriptive character sequence. When the intermediate image processing model processes the adjusted sample image H times to obtain H sets of character prediction results, any character in these H sets of character prediction results is determined based on the probability of the 1 set of predicted result characters corresponding to that character using a preset rule. This 1 set of predicted result characters consists of characters with a non-zero probability among the multiple candidate characters predicted by the intermediate image processing model for the adjusted sample image. The preset rules can be set according to different adjustment needs. For example, the preset rules can indicate that any character in the H group of character prediction results is randomly determined based on the probability of the 1 group of result prediction characters corresponding to that character; or, any character in the h-th group of character prediction results in the H group of character prediction results is the result prediction character corresponding to the h-th probability when the probabilities of the 1 group of result prediction characters corresponding to that character are sorted from largest to smallest; where h is a positive integer less than or equal to H.

[0082] Furthermore, based on the prediction results of the H group of characters and the gradient function, the image processing device optimizes and adjusts the intermediate image processing model to obtain the final image processing model. The gradient function can be given by Equation 2:

[0083]

[0084] Where H is the number of character prediction result sets, h is the independent variable representing the number of character prediction result sets, and Z is the length of a character prediction result set (i.e., the number of characters contained in a character prediction result set). θ2 represents the prediction result of the h-th character group; I2 is used to indicate the visual features of the adjusted sample image; θ2 is the model parameters used by the intermediate image processing model when processing the adjusted sample image; R is the scoring function, for example, it can be a scoring function based on the CIDEr metric (full name: Consensus-based Image Description Evaluation); b is the baseline score.

[0085] In one embodiment, based on evaluation metrics widely used in the field of image description to evaluate models that implement image description functions, the image processing model proposed in this application embodiment is evaluated and compared with existing models for implementing image description functions. It is found that the image processing model proposed in this application embodiment has a significant improvement in the speed of implementing image description functions (i.e., a significant improvement in the speed of generating descriptive text to describe the image content), and also possesses good generation quality (i.e., the generated descriptive text to describe the image content has good text quality). Furthermore, the image processing model proposed in this application embodiment is easy to train. As shown in Table 1, this application embodiment is based on the MSCOCO dataset, which is widely used in the field of image description. The evaluation metrics widely used in the field of image description are adopted: language evaluation metrics for evaluating generation quality, including BLEU (Bilingual Evaluation Understudy), METEOR (Metric for Evaluation of Translation with Explicit Ordering), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), SPICE (Semantic Propositional Image Caption Evaluation), and CIDEr; latency metrics for evaluating generation rate; and speedup metrics for evaluating model algorithms. Among them, the BLEU metric can include BLEU-1 and BLEU-4 depending on the specific calculation method.The image processing model proposed in the embodiments of this application (taking K=4 as an example) was compared and evaluated with existing models, including: Autoregressive Image Captioning (AIC), which belongs to the Autoregressive Model; Fast Non-Autoregressive Image Captioning (FNIC), which belongs to the Non-Autoregressive Model; Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning (CMAL), which belongs to the Non-Autoregressive Model; and Partially Non-Autoregressive Image Captioning (PNAIC), which belongs to the Partially Non-Autoregressive Model.

[0086] Table 1

[0087]

[0088] The comparison results show that the image processing model proposed in this application has a significantly shorter generation latency compared to the AIC model in the automatic regression model, which means a significant improvement in generation rate; compared to the FNIC and CMAL models in the non-automatic regression model, it has a certain improvement in generation quality; and compared to the PNAIC model in the semi-automatic regression model, it is superior in model algorithm.

[0089] In this embodiment, the image processing device can train an initial image processing model based on training samples including training sample images and target description character sequences of the training sample images to obtain an image processing model. Specifically, it can take the K start characters included in the label description character sequence of the training sample image as a reference, and increment by M to sequentially truncate the label description character sequence of the training sample image to obtain P training description character sequences corresponding to the training sample image. Then, the training sample image is used as the input of the initial image processing model, and the P training description character sequences are used as the input of the initial image processing model in sequence, and the target description character sequence of the training sample image is used as the expected output to train the initial image processing model to optimize and obtain the image processing model. Among them, each training description character sequence in the P training description character sequences includes M training description characters, M and P are integers greater than 1, and K is a positive integer less than or equal to M. This allows the trained image processing model to predict N descriptive character sequences corresponding to the image to be processed, based on the visual feature vector and K start characters of the image. Each of the N descriptive character sequences includes M descriptive characters. This reduces the number of times the image processing model processes the visual feature vector of the image to be processed, thus improving the speed at which the image processing model generates predicted descriptive character sequences to describe the image content. Furthermore, the intermediate image processing model obtained after training the initial image processing model can be optimized and adjusted based on the gradient function to obtain a higher quality image processing model that generates predicted descriptive character sequences to describe the image content.

[0090] Based on the above-described image processing method embodiments, this application provides an image processing apparatus. See also... Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. The image processing device may include an acquisition unit 1001 and a processing unit 1002. Figure 10 The image processing apparatus shown can operate the following units:

[0091] The processing unit 1002 is used to perform visual feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed.

[0092] The processing unit 1002 is further configured to perform character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences; wherein, each of the N descriptive character sequences includes M descriptive characters, the first descriptive character sequence of the N descriptive character sequences is obtained based on the visual feature vector of the image to be processed, the (t+1)th descriptive character sequence of the N descriptive character sequences is obtained based on the previous t descriptive character sequences, M and N are integers greater than 1, and t is a positive integer less than or equal to N;

[0093] The processing unit 1002 is further configured to generate a predicted descriptive character sequence for the image to be processed based on the N descriptive character sequences.

[0094] In one embodiment, when the processing unit 1002 performs character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences, it specifically performs the following operations:

[0095] The visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed are subjected to character prediction processing to obtain the first descriptive character sequence; the first reference descriptive character sequence includes K start characters, and the M descriptive characters included in the first descriptive character sequence are generated in parallel, where K is a positive integer less than or equal to M;

[0096] The visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first descriptive character sequence are subjected to character prediction processing to obtain a second descriptive character sequence; the M descriptive characters included in the second descriptive character sequence are generated in parallel.

[0097] In one embodiment, when the processing unit 1002 performs visual feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed, it specifically performs the following operations:

[0098] The image to be processed is processed by a region-based fast convolutional neural network in the image processing model to extract regional object features and obtain the regional object feature vector of the image to be processed.

[0099] The encoder in the image processing model performs context feature extraction on the regional object feature vectors of the image to be processed, thereby obtaining the visual feature vectors of the image to be processed.

[0100] In one embodiment, when the processing unit 1002 performs character prediction processing on the visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed to obtain the first descriptive character sequence, it specifically performs the following operations:

[0101] The visual feature vector of the image to be processed and the first reference description character sequence are subjected to feature interaction processing to obtain the prediction vector corresponding to the first reference description character sequence.

[0102] The prediction vector corresponding to the first reference description character sequence is subjected to character probability prediction processing to obtain the probabilities of M groups of predicted characters; the probability of each group of predicted characters in the M groups of predicted characters includes: the probability corresponding to each predicted character in each group of predicted characters.

[0103] Based on the probabilities of the M groups of predicted characters, a descriptive character is determined from each group of predicted characters to obtain the first descriptive character sequence.

[0104] In one embodiment, when the processing unit 1002 performs feature interaction processing on the visual feature vector of the image to be processed and the first reference description character sequence to obtain the prediction vector corresponding to the first reference description character sequence, it specifically performs the following operations:

[0105] The first reference description character sequence is subjected to feature embedding processing through the embedding layer in the image processing model to obtain the embedding vector corresponding to the first reference description character sequence.

[0106] The embedding vector corresponding to the first reference description character sequence is subjected to feature masking processing through the multi-head self-attention layer in the image processing model to obtain the masking vector corresponding to the first reference description character sequence.

[0107] The multi-head cross-domain attention layer in the image processing model performs attention interaction processing on the visual feature vector of the image to be processed and the masking vector corresponding to the first reference description character sequence to obtain the interaction vector corresponding to the first reference description character sequence.

[0108] The interaction vector corresponding to the first reference description character sequence is processed by the forward fully connected layer in the image processing model to obtain the prediction vector corresponding to the first reference description character sequence.

[0109] In one embodiment, when the processing unit 1002 determines the descriptive character from each group of predicted characters based on the probabilities of the M groups of predicted characters to obtain the first descriptive character sequence, it specifically performs the following operations:

[0110] The predicted character with the highest probability in each group of predicted characters is determined as the description character corresponding to each group of predicted characters, thus obtaining the first description character sequence.

[0111] In one embodiment, the image processing model is obtained by training an initial image processing model;

[0112] The acquisition unit 1001 is used to acquire training samples, the training samples including: training sample images and target description character sequences of the training sample images;

[0113] The processing unit 1002 is further configured to, based on the K start characters included in the label description character sequence of the training sample image, and with an increment of M, sequentially truncate the label description character sequence of the training sample image to obtain P training description character sequences corresponding to the training sample image; the label description character sequence of the training sample image is composed of the K start characters and the target description character sequence of the training sample image, where P is an integer greater than 1;

[0114] The processing unit 1002 is further configured to use the training sample image as input to the initial image processing model, and sequentially use the P training descriptive character sequences as input to the initial image processing model, and use the target descriptive character sequence of the training sample image as the expected output to train the initial image processing model to optimize and obtain the image processing model.

[0115] In one embodiment, the image processing model is obtained by optimizing and adjusting an intermediate image processing model obtained after training the initial image processing model;

[0116] The acquisition unit 1001 is also used to acquire the adjusted sample image;

[0117] The processing unit 1002 is further configured to use the adjusted sample image as input to the intermediate image processing model to obtain H sets of character prediction results; wherein, each set of character prediction results in the H sets of character prediction results includes: Q result description character sequences, each result description character sequence in the Q sets of result description character sequences includes M result description characters, the (z+1)th result description character sequence in the Q sets of result description character sequences is obtained based on the previous z result description character sequences, H and Q are integers greater than 1, and z is a positive integer less than or equal to Q;

[0118] The processing unit 1002 is further configured to optimize and adjust the intermediate image processing model based on the prediction results of the H group characters and the gradient function to obtain the image processing model.

[0119] According to one embodiment of this application, Figure 1 , Figure 5 as well as Figure 7 The image processing method shown can involve various steps that can be derived from... Figure 10 This is performed by the individual units in the image processing apparatus shown. For example, Figure 1 Steps S101 to S103 shown can be derived from... Figure 10 The image processing unit 1002 in the illustrated image processing apparatus performs the operation. For example, Figure 5 Steps S501 to S505 shown can be derived from... Figure 10 The image processing unit 1002 in the illustrated image processing apparatus performs the operation. For example, Figure 7 The step S701 shown can be performed by Figure 10 The image processing apparatus shown is executed by the acquisition unit 1001. Figure 7 Steps S702 to S703 shown can be derived from... Figure 10 The processing unit 1002 in the image processing apparatus shown performs the operation.

[0120] According to another embodiment of this application, Figure 10 The various units in the illustrated image processing apparatus can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image processing apparatus based on logical function division may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0121] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 1 , Figure 5 as well as Figure 7 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 10 The image processing apparatus shown herein, and the image processing method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and run therein.

[0122] In this embodiment, the processing unit 1002 can perform visual feature extraction processing on the image to be processed to obtain a visual feature vector of the image to be processed; then, it can perform character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences; and generate a predicted descriptive character sequence of the image to be processed based on the N descriptive character sequences. Each of the N descriptive character sequences includes M descriptive characters, the first descriptive character sequence is obtained from the visual feature vector of the image to be processed, and the (t+1)th descriptive character sequence is obtained from the previous t descriptive character sequences. M and N are integers greater than 1, and t is a positive integer less than or equal to N. That is, the visual feature vector of the image to be processed can be processed multiple times to obtain multiple descriptive character sequences; then, a predicted descriptive character sequence for describing the image content of the image to be processed can be generated based on the obtained multiple descriptive character sequences. Each processing of the visual feature vector of the image to be processed yields a descriptive character sequence including multiple descriptive characters, which can reduce the number of processing steps for the visual feature vector of the image to be processed and improve the generation rate of the predicted descriptive character sequence for describing the image content.

[0123] Based on the above-described image processing method and apparatus embodiments, this application also provides an image processing device. See also... Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Figure 11 The image processing device shown may include at least a processor 1101, an input interface 1102, an output interface 1103, and a computer storage medium 1104. The processor 1101, input interface 1102, output interface 1103, and computer storage medium 1104 may be connected via a bus or other means.

[0124] Computer storage medium 1104 can be stored in the memory of the image processing device. Computer storage medium 1104 is used to store computer programs, which include program instructions. Processor 1101 is used to execute the program instructions stored in computer storage medium 1104. Processor 1101 (or CPU (Central Processing Unit)) is the computing and control core of the image processing device. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to realize the above-mentioned image processing method flow or corresponding functions.

[0125] This application embodiment also provides a computer storage medium (Memory), which is a memory device in an image processing device used to store programs and data. It is understood that the computer storage medium here can include the built-in storage medium in the terminal, or it can include an extended storage medium supported by the terminal. The computer storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1101. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium here can be a high-speed random access memory (RAM), or it can be non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor.

[0126] In one embodiment, the processor 1101 and the input interface 1102 may load and execute one or more instructions stored in the computer storage medium to achieve the aforementioned related... Figure 1 , Figure 5 as well as Figure 7 In the corresponding steps of the image processing method embodiment, in a specific implementation, one or more instructions in the computer storage medium are loaded and executed by the processor 1101 as follows:

[0127] Visual feature extraction is performed on the image to be processed to obtain the visual feature vector of the image to be processed;

[0128] The visual feature vector of the image to be processed is subjected to character prediction processing to obtain N descriptive character sequences; wherein, each of the N descriptive character sequences includes M descriptive characters, the first descriptive character sequence of the N descriptive character sequences is obtained based on the visual feature vector of the image to be processed, the (t+1)th descriptive character sequence of the N descriptive character sequences is obtained based on the previous t descriptive character sequences, M and N are integers greater than 1, and t is a positive integer less than or equal to N;

[0129] Generate a predicted descriptive character sequence for the image to be processed based on the N descriptive character sequences.

[0130] In one embodiment, when the processor 1101 performs character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences, it specifically performs the following operations:

[0131] The visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed are subjected to character prediction processing to obtain the first descriptive character sequence; the first reference descriptive character sequence includes K start characters, and the M descriptive characters included in the first descriptive character sequence are generated in parallel, where K is a positive integer less than or equal to M;

[0132] The visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first descriptive character sequence are subjected to character prediction processing to obtain a second descriptive character sequence; the M descriptive characters included in the second descriptive character sequence are generated in parallel.

[0133] In one embodiment, when the processor 1101 performs visual feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed, it specifically performs the following operations:

[0134] The image to be processed is processed by a region-based fast convolutional neural network in the image processing model to extract regional object features and obtain the regional object feature vector of the image to be processed.

[0135] The encoder in the image processing model performs context feature extraction on the regional object feature vectors of the image to be processed, thereby obtaining the visual feature vectors of the image to be processed.

[0136] In one embodiment, when the processor 1101 performs character prediction processing on the visual feature vector of the image to be processed and the first reference descriptive character sequence corresponding to the image to be processed to obtain the first descriptive character sequence, the following operations are specifically performed:

[0137] The visual feature vector of the image to be processed and the first reference description character sequence are subjected to feature interaction processing to obtain the prediction vector corresponding to the first reference description character sequence.

[0138] The prediction vector corresponding to the first reference description character sequence is subjected to character probability prediction processing to obtain the probabilities of M groups of predicted characters; the probability of each group of predicted characters in the M groups of predicted characters includes: the probability corresponding to each predicted character in each group of predicted characters.

[0139] Based on the probabilities of the M groups of predicted characters, a descriptive character is determined from each group of predicted characters to obtain the first descriptive character sequence.

[0140] In one embodiment, when the processor 1101 performs feature interaction processing on the visual feature vector of the image to be processed and the first reference description character sequence to obtain the prediction vector corresponding to the first reference description character sequence, it specifically performs the following operations:

[0141] The first reference description character sequence is subjected to feature embedding processing through the embedding layer in the image processing model to obtain the embedding vector corresponding to the first reference description character sequence.

[0142] The embedding vector corresponding to the first reference description character sequence is subjected to feature masking processing through the multi-head self-attention layer in the image processing model to obtain the masking vector corresponding to the first reference description character sequence.

[0143] The multi-head cross-domain attention layer in the image processing model performs attention interaction processing on the visual feature vector of the image to be processed and the masking vector corresponding to the first reference description character sequence to obtain the interaction vector corresponding to the first reference description character sequence.

[0144] The interaction vector corresponding to the first reference description character sequence is processed by the forward fully connected layer in the image processing model to obtain the prediction vector corresponding to the first reference description character sequence.

[0145] In one embodiment, when the processor 1101 determines the descriptive character from each group of predicted characters based on the probabilities of the M groups of predicted characters to obtain the first descriptive character sequence, it specifically performs the following operations:

[0146] The predicted character with the highest probability in each group of predicted characters is determined as the description character corresponding to each group of predicted characters, thus obtaining the first description character sequence.

[0147] In one embodiment, the image processing model is obtained by training an initial image processing model;

[0148] The input interface 1102 is used to acquire training samples, the training samples including: training sample images and target description character sequences of the training sample images;

[0149] The processor 1101 is further configured to, based on the K start characters included in the label description character sequence of the training sample image, and with an increment of M, sequentially truncate the label description character sequence of the training sample image to obtain P training description character sequences corresponding to the training sample image; the label description character sequence of the training sample image is composed of the K start characters and the target description character sequence of the training sample image, where P is an integer greater than 1;

[0150] The processor 1101 is further configured to use the training sample image as input to the initial image processing model, and sequentially use the P training descriptive character sequences as input to the initial image processing model, and use the target descriptive character sequence of the training sample image as the expected output to train the initial image processing model to optimize and obtain the image processing model.

[0151] In one embodiment, the image processing model is obtained by optimizing and adjusting an intermediate image processing model obtained after training the initial image processing model;

[0152] The input interface 1102 is also used to acquire the adjustment sample image;

[0153] The processor 1101 is further configured to use the adjusted sample image as input to the intermediate image processing model to obtain H sets of character prediction results; wherein, each set of character prediction results in the H sets of character prediction results includes: Q result description character sequences, each result description character sequence in the Q sets of result description character sequences includes M result description characters, the (z+1)th result description character sequence in the Q sets of result description character sequences is obtained based on the previous z result description character sequences, H and Q are integers greater than 1, and z is a positive integer less than or equal to Q;

[0154] The processor 1101 is further configured to optimize and adjust the intermediate image processing model based on the prediction results of the H group characters and the gradient function to obtain the image processing model.

[0155] This application provides a computer program product or computer program, the computer program product including a computer program stored in a computer storage medium; the processor of an image processing device reads the computer program from the computer storage medium, and the processor executes the computer program, causing the image processing device to perform the above-described actions. Figure 1 , Figure 5 as well as Figure 7 The method embodiment shown. The computer-readable storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0156] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start> < / start>

Claims

1. An image processing method, characterized in that, include: Acquire the adjusted sample image; The adjusted sample image is used as input to the intermediate image processing model to obtain H sets of character prediction results; wherein, the intermediate image processing model is obtained by training the initial image processing model, and each set of character prediction results in the H sets of character prediction results includes: Q result description character sequences, each of the Q result description character sequences includes M result description characters, and the (z+1)th result description character sequence in the Q result description character sequences is obtained based on the previous z result description character sequences, where H and Q are integers greater than 1, and z is a positive integer less than or equal to Q; Based on the character prediction results and gradient function of the H group, the intermediate image processing model is optimized and adjusted to obtain the image processing model; The image processing model is used to extract visual features from the image to be processed, thereby obtaining the visual feature vector of the image to be processed. The image processing model performs character prediction processing on the visual feature vector of the image to be processed to obtain N descriptive character sequences; wherein, each of the N descriptive character sequences includes M descriptive characters, the M descriptive characters included in each descriptive character sequence are generated in parallel, the first descriptive character sequence of the N descriptive character sequences is obtained based on the visual feature vector of the image to be processed, the (t+1)th descriptive character sequence of the N descriptive character sequences is obtained based on the previous t descriptive character sequences, M and N are integers greater than 1, and t is a positive integer less than or equal to N; The image processing model generates a predicted descriptive character sequence for the image to be processed based on the N descriptive character sequences. The first descriptive character sequence is obtained based on the visual feature vector of the image to be processed, including: The visual feature vector of the image to be processed and the first reference description character sequence corresponding to the image to be processed are subjected to feature interaction processing to obtain the prediction vector corresponding to the first reference description character sequence. The first reference description character sequence includes K start characters, where K is a positive integer less than or equal to M. The prediction vector corresponding to the first reference description character sequence is subjected to character probability prediction processing to obtain the probabilities of M groups of predicted characters; the probability of each group of predicted characters in the M groups of predicted characters includes: the probability corresponding to each predicted character in each group of predicted characters. Based on the probabilities of the M groups of predicted characters, the predicted character indicated by the highest probability in each group of predicted characters is determined as the description character corresponding to each group of predicted characters, thus obtaining the first description character sequence.

2. The method as described in claim 1, characterized in that, The visual feature vector of the image to be processed is subjected to character prediction processing to obtain N descriptive character sequences, including: The visual feature vector of the image to be processed, the first reference descriptive character sequence, and the first descriptive character sequence are subjected to character prediction processing to obtain a second descriptive character sequence; the M descriptive characters included in the second descriptive character sequence are generated in parallel.

3. The method as described in claim 1, characterized in that, The step of performing visual feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed includes: The image processing model uses a region-based fast convolutional neural network to extract regional object features from the image to be processed, thereby obtaining the regional object feature vector of the image to be processed. The encoder in the image processing model performs context feature extraction on the regional object feature vectors of the image to be processed, thereby obtaining the visual feature vectors of the image to be processed.

4. The method as described in claim 1, characterized in that, The step of performing feature interaction processing on the visual feature vector of the image to be processed and the first reference description character sequence corresponding to the image to be processed to obtain the prediction vector corresponding to the first reference description character sequence includes: The first reference description character sequence is subjected to feature embedding processing through the embedding layer in the image processing model to obtain the embedding vector corresponding to the first reference description character sequence. The embedding vector corresponding to the first reference description character sequence is subjected to feature masking processing through the multi-head self-attention layer in the image processing model to obtain the masking vector corresponding to the first reference description character sequence. The multi-head cross-domain attention layer in the image processing model performs attention interaction processing on the visual feature vector of the image to be processed and the masking vector corresponding to the first reference description character sequence to obtain the interaction vector corresponding to the first reference description character sequence. The interaction vector corresponding to the first reference description character sequence is processed by the forward fully connected layer in the image processing model to obtain the prediction vector corresponding to the first reference description character sequence.

5. The method as described in claim 1, characterized in that, The image processing model is obtained by training an initial image processing model; the method further includes: Obtain training samples, which include: training sample images and target description character sequences of the training sample images; Using the K start characters included in the label description character sequence of the training sample image as a reference, and with an increment of M, the label description character sequence of the training sample image is truncated sequentially to obtain P training description character sequences corresponding to the training sample image; the label description character sequence of the training sample image is composed of the K start characters and the target description character sequence of the training sample image, where P is an integer greater than 1; The training sample image is used as the input to the initial image processing model, and the P training descriptive character sequences are used as the input to the initial image processing model in sequence. The target descriptive character sequence of the training sample image is used as the expected output to train the initial image processing model and optimize it to obtain the image processing model.

6. An image processing apparatus, characterized in that, include: The image processing apparatus includes an acquisition unit and a processing unit, wherein the acquisition unit and the processing unit are used to perform the image processing method as described in any one of claims 1-5.

7. An image processing device, characterized in that, The image processing device includes an input interface and an output interface, and further includes: A processor, adapted to implement one or more instructions; and, A computer storage medium storing one or more instructions adapted to be loaded by the processor and executed as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, are used to perform the image processing method as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, is used to load and execute the image processing method as described in any one of claims 1-5.

Images