Method and system for generating image description information, electronic device, and storage medium

Abstract

The present invention discloses a method for generating image description information, a system thereof, an electronic device, and a storage medium, and relates to the field of artificial intelligence. The method comprises: receiving an image to be processed, segmenting the image to be processed to obtain multiple segmented sub-images, constructing an initial feature vector based on the sub-images and image sequence information corresponding to each sub-image, injecting preset weight information into the initial feature vector to obtain a weighted feature vector, decoding the weighted feature vector to generate a candidate information sequence, parsing the candidate information sequence to obtain multiple image description information, and selecting the image description information represented by the highest evaluation score as the target image description information for the image to be processed. The present invention solves the technical problems in related technologies of logical confusion in generated image description information and ambiguous sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for generating image description information, a system thereof, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning technology is providing intelligent innovation scenarios for various industries, such as image description generation scenarios, by relying on its complex neural network structure, learning ability that can continuously improve its performance as the data scale increases, and technical advantages such as the ability to directly extract features from massive data.

[0003] In related technologies, the encoder-decoder model framework is often used as the underlying architecture for constructing description generation models for image description generation. This task consists of two parts: the encoder and the decoder. A CNN (convolutional neural network) is used to parse the input image, identify objects, and encode the image information. The decoder then uses a recurrent neural network (RNN)-based model (such as LSTM (Long Short-Term Memory) and BiLSTM (Bidirectional Long Short-Term Memory)) to decode the image encoding and generate the corresponding text description.

[0004] Although current image description methods can express the behavioral and object information contained in images, they have the following defects: (1) Sentiment analysis ambiguity: Since most adjectives do not have polarity and only appear polarity when paired with nouns, there will be logical ambiguity. For example, for the adjective "big", "big screen" tends to be a positive description, but "loud noise" tends to be a negative description; for the adjective "high", "high cost performance" is a positive description, but "high oil prices" is a negative description, etc. (2) Semantic confusion and loss: Since a single image lacks contextual information reference, the generation of image descriptions often results in the reversal of primary and secondary objects and the misattribution of actions. For example, illogical and incorrect descriptions such as a pony riding a person, the sky in the ocean, and a person walking in a horse's belly.

[0005] Therefore, there is an urgent need to find an effective solution to the problem that the generated text descriptions need to be repeatedly modified due to the chaotic logic of the text and the translation sentences not conforming to the language grammatical rules, resulting in low efficiency and low accuracy.

[0006] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0007] The embodiments of the present invention provide a method for generating image description information and a system thereof, an electronic device, and a storage medium, so as to at least solve the technical problems in related technologies of logical confusion in the generated image description information and ambiguous sentiment analysis.

[0008] According to one aspect of an embodiment of the present invention, a method for generating image description information is provided, comprising: receiving an image to be processed, and segmenting the image to be processed to obtain a plurality of segmented sub-images; constructing an initial feature vector based on the sub-images and image sequence information corresponding to each of the sub-images; injecting preset weight information into the initial feature vector to obtain a weighted feature vector; decoding the weighted feature vector to generate a candidate information sequence, and parsing the candidate information sequence to obtain a plurality of image description information, wherein each image description information corresponds to an information evaluation score; and selecting the image description information represented by the highest evaluation score as the target image description information of the image to be processed.

[0009] Optionally, after receiving the image to be processed, the method further includes: obtaining preset image information, wherein the preset image information includes at least one of the following: color, boundary, and grayscale; and pre-processing the image to be processed based on the preset image information.

[0010] Optionally, the step of segmenting the image to be processed to obtain multiple segmented sub-images includes: identifying the image to be processed to obtain target information, wherein the target information includes at least one of the following: an object, a scene; based on the target information, segmenting the image to be processed to obtain multiple segmented sub-images.

[0011] Optionally, after injecting preset weight information into the initial feature vector to obtain the weight feature vector, it also includes: when the image to be processed is an intermediate image in a continuous image set, receiving the encoding information of the previous image, wherein the intermediate image is any image in the continuous image set except the start image, and the previous image is an image at a time before the time of the image to be processed in the time series of the continuous image set; injecting the encoding information into the weight feature vector to obtain the target weight feature vector.

[0012] Optionally, after injecting the encoded information into the weight feature vector to obtain the target weight feature vector, it also includes: based on a preset dimensionality reduction strategy, reducing the dimension of the target weight feature vector to obtain a reduced dimensionality feature vector; judging whether the reduced dimensionality feature vectors have a similar classification relationship to obtain a judgment result, wherein the similar classification relationship refers to the existence of image description information with equal weight values; when the judgment result indicates that the reduced dimensionality feature vectors have a similar classification relationship, the image description information in the previous image is used as the image description information in the image to be processed.

[0013] Optionally, after parsing the candidate information sequence to obtain multiple image description information, it also includes: based on a preset evaluation strategy, evaluating the multiple image description information to obtain an information evaluation score corresponding to each image description information; sorting the information evaluation scores to obtain a sorting result; based on the sorting result, selecting all the information evaluation scores located before a preset number of bits as target evaluation scores; when the image to be processed is not the last image in the continuous image set, passing the encoded information of the image description information indicated by the target evaluation score into the image description information generation process of the next image.

[0014] According to another aspect of an embodiment of the present invention, a system for generating image description information is further provided, including: an image information parsing module, configured to receive an image to be processed, and segment the image to be processed to obtain a plurality of segmented sub-images, construct an initial feature vector based on the sub-images and image sequence information corresponding to each sub-image, and transmit the initial feature vector to an image information encoding module; an image information encoding module, connected to the image information parsing module, configured to receive the initial feature vector, and inject preset weight information into the initial feature vector to obtain a weight feature vector, and transmit the weight feature vector to an information encoding processing module; a grammar checking and evaluation module, connected to the image information encoding module and the image description generating module, and configured to transmit the weight feature vector to the image information encoding module. Preset weight information, and receive the image description information transmitted by the image description generation module, evaluate the image description information, and return the obtained information evaluation score to the image description generation module; an information coding processing module, connected to the image information coding module, for receiving the weight feature vector, and decoding the weight feature vector, generating a candidate information sequence, and transmitting the candidate information sequence to the image description generation module; an image description generation module, connected to the information coding processing module, for receiving the candidate information sequence, and parsing the candidate information sequence to obtain multiple image description information, and selecting the image description information represented by the highest evaluation score as the target image description information of the image to be processed, wherein each of the image description information corresponds to an information evaluation score.

[0015] Optionally, the grammar check evaluation module includes: a weight information processing unit for receiving the preset weight information, wherein the preset weight information includes at least: action weight information, position weight information, and master-slave weight information; an inheritance information receiving unit for receiving the encoding information of the previous image when the image to be processed is an intermediate image in a continuous image set, and injecting the encoding information into the weight feature vector, wherein the intermediate image is any image in the continuous image set except the start image, and the previous image is an image at a previous time relative to the time of the image to be processed in the time series of the continuous image set; an image description evaluation unit for evaluating the image description information based on a preset evaluation strategy to obtain an information evaluation score corresponding to each image description information.

[0016] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for generating image description information.

[0017] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for generating image description information.

[0018] In the present disclosure, an image to be processed is received and segmented to obtain a plurality of segmented sub-images. An initial feature vector is constructed based on the sub-images and the image sequence information corresponding to each sub-image. Preset weight information is injected into the initial feature vector to obtain a weight feature vector. The weight feature vector is decoded to generate a candidate information sequence. The candidate information sequence is parsed to obtain a plurality of image description information. The image description information represented by the highest evaluation score is selected as the target image description information of the image to be processed. In the present application, the weight feature vector can be constructed by injecting preset weight information into the obtained initial feature vector. The weight feature vector is then decoded to generate a candidate information sequence. The candidate information sequence is parsed to obtain a plurality of image description information. The image description information is then evaluated and the image description information with the highest evaluation score is selected as the target image description information of the image to be processed. This can not only effectively avoid the defects of logical confusion and sentiment analysis ambiguity in the generated image description information, but also improve the generation efficiency and generation accuracy, thereby solving the technical problems in the related art that the generated image description information has logical confusion and sentiment analysis is prone to ambiguity. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a flowchart of an optional method for generating image description information according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of an optional system for generating image description information according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of an optional structure of an image information parsing module 1 according to an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of an optional structure of an image information encoding module 2 according to an embodiment of the present invention;

[0024] Figure 5 is a schematic diagram of an optional structure of a grammar checking and evaluation module 3 according to an embodiment of the present invention;

[0025] Figure 6 is a schematic diagram of an optional information encoding processing module 4 structure according to an embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of an optional structure of an image description generating module 5 according to an embodiment of the present invention;

[0027] Figure 8 is a schematic diagram of an optional image description generation process according to an embodiment of the present invention;

[0028] Figure 9 This is a hardware structure block diagram of an electronic device (or mobile device) for a method for generating image description information according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] To facilitate those skilled in the art to understand the present invention, some of the terms or nouns involved in the embodiments of the present invention are explained below:

[0032] Convolutional Neural Networks (CNNs) are a type of feedforward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. CNNs possess representational learning capabilities, enabling translation-invariant classification of input information based on their hierarchical structure.

[0033] A recurrent neural network (RNN) is a type of recursive neural network that takes sequence data as input, performs recursion in the direction of sequence evolution, and connects all nodes (recurrent units) in a chain-like manner.

[0034] Long Short-Term Memory (LSTM) is a time-recurrent neural network that solves the problems of general RNNs. General RNNs have a repetitive neural network module connected in a chain.

[0035] Bidirectional Long Short-Term Memory (BiLSTM) neural network, which is composed of a forward LSTM and a backward LSTM, is used for deep encoding information extraction.

[0036] The attention mechanism is a technology that allows the model to focus on important information and fully learn and extract the information. The attention mechanism is applicable to any sequence model.

[0037] The encoder-decoder network model is a common model framework in deep learning. The encoder and decoder parts can be arbitrary text, speech, images, and videos. The model can adopt various structures such as CNN, RNN, BRNN (bidirectional recurrent neural network) and GRU (recurrent neural network), providing an end-to-end learning framework for various network models.

[0038] High-dimensional data visualization: When analyzing high-dimensional data, data dimensionality reduction algorithms are used because the data cannot be clearly classified in high-dimensional space, so that the characteristic distribution of the data in low-dimensional space can be clearly seen.

[0039] Bilingual Evaluation Understudy (BLEU) is an evaluation metric for machine translation results, with the overall concept being accuracy.

[0040] Image Caption allows the algorithm to automatically generate corresponding descriptive text based on an input image or video. The descriptive text is a complete paragraph or sentence to describe what is happening in the current image, the objects that exist, etc.

[0041] It should be noted that the method for generating image description information and the system thereof in the present disclosure can be used in the field of artificial intelligence when generating image description information, and can also be used in any field other than the field of artificial intelligence when generating image description information. The present disclosure does not limit the application field of the method for generating image description information and the system thereof.

[0042] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display and analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information provided by the aforementioned user or organization.

[0043] The following embodiments of the present invention can be applied to various systems, applications, and devices that generate image description information. Image description generation in these embodiments, as an emerging application of deep learning in image understanding and dynamic object detection, has technical applications in areas such as news release generation, video subtitle processing, and assistive reading for people with disabilities (for example, helping deaf-mute people watch movies).

[0044] The present invention uses a weighting information injection mechanism to weight common logical actions, scenes, and locations in life (for example, "a person riding a horse" is more plausible than "a horse riding a person," so "a person riding a horse" has a higher weight; "the sun is above the horizon" is more likely to occur than "the sun is below the horizon," so "the sun is above the horizon" has a higher weight). Injecting this weighting information into the coded information effectively prevents illogical image description generation. A coding inheritance mechanism provides a contextual reference for tasks such as video and a series of consecutive image descriptions. By using the previous coded information as a reference for the current description generation, it resolves the hierarchy issues that the weighting information injection mechanism cannot address (for example, if the image description is "a boy picks up a girl," but "a girl picks up a boy" also has the same weight, but the inherited coded information indicates that it is "a boy picks up a girl," the weighting of the current image's picking action will be biased towards the coded result of "a boy picks up a girl." In other words, if the present invention discovers a similar classification relationship through vector space dimensionality reduction, it determines that the current image description is similar to the previous one). The generated result is then inherited in the next image description task.

[0045] Through the evaluation mechanism, the generated description information can be scored using a preset evaluation method (for example, BLEU evaluation), so that the quality of the generated information can be guaranteed. In addition, the reference library of this evaluation method can be manually imported and accumulated in its own cycle, providing self-learning capabilities for the entire image description generation system, thereby ensuring the quality of the generated description information.

[0046] Based on the above mechanism, the present invention can make the information generated by image description have higher quality and be more in line with natural language grammatical rules, and can effectively reduce the number of manual revisions, thereby improving the efficiency and accuracy of image description information generation.

[0047] The present invention will be described in detail below with reference to various embodiments. Example 1

[0048] According to an embodiment of the present invention, an embodiment of a method for generating image description information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Figure 1 is a flowchart of an optional method for generating image description information according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0050] Step S101 : receiving an image to be processed and segmenting the image to be processed to obtain a plurality of segmented sub-images.

[0051] Step S102 : constructing an initial feature vector based on the sub-images and image sequence information corresponding to each sub-image.

[0052] Step S103: injecting preset weight information into the initial feature vector to obtain a weighted feature vector.

[0053] Step S104 : decoding the weighted feature vector to generate a candidate information sequence, and parsing the candidate information sequence to obtain a plurality of image description information, wherein each image description information corresponds to an information evaluation score.

[0054] Step S105 : selecting the image description information represented by the highest evaluation score as the target image description information of the image to be processed.

[0055] Through the above steps, an image to be processed can be received and segmented to obtain multiple segmented sub-images. Based on the sub-images and the image sequence information corresponding to each sub-image, an initial feature vector is constructed, preset weight information is injected into the initial feature vector to obtain a weight feature vector, the weight feature vector is decoded, a candidate information sequence is generated, and the candidate information sequence is parsed to obtain multiple image description information, and the image description information represented by the highest evaluation score is selected as the target image description information of the image to be processed. In an embodiment of the present invention, the preset weight information can be injected into the obtained initial feature vector to construct a weight feature vector, and then the weight feature vector is decoded to generate a candidate information sequence, and the candidate information sequence is parsed to obtain multiple image description information. After that, by evaluating the image description information, the image description information with the highest evaluation score is selected as the target image description information of the image to be processed. This can not only effectively avoid the defects of logical confusion and sentiment analysis ambiguity in the generated image description information, but also improve the generation efficiency and generation accuracy, thereby solving the technical problems of logical confusion and sentiment analysis ambiguity in the generated image description information in the related art.

[0056] The embodiment of the present invention is described in detail below in conjunction with the above steps.

[0057] Step S101 : receiving an image to be processed and segmenting the image to be processed to obtain a plurality of segmented sub-images.

[0058] Optionally, the step of segmenting the image to be processed to obtain multiple segmented sub-images includes: identifying the image to be processed to obtain target information, wherein the target information includes at least one of the following: an object, a scene; and segmenting the image to be processed based on the target information to obtain multiple segmented sub-images.

[0059] In an embodiment of the present invention, an image to be processed can be received by an image information analysis module, and then the image to be processed is identified. Based on the identified target information (including: objects, scenes, etc.), the objects, scenes and other information in the image to be processed are segmented to obtain multiple segmented sub-images.

[0060] Optionally, after receiving the image to be processed, the method further includes: obtaining preset image information, wherein the preset image information includes at least one of the following: color, boundary, and grayscale; and pre-processing the image to be processed based on the preset image information.

[0061] In an embodiment of the present invention, preset image information (including color, boundary, grayscale and other information) can be obtained first according to the recognition requirements of the model to be used, and then the image to be processed is preprocessed according to the preset image information, that is, the image to be processed is cropped according to the color, boundary, grayscale and other information, and the image information such as grayscale or color is extracted so that the image to be processed meets the recognition requirements of the model to be used.

[0062] Step S102 : constructing an initial feature vector based on the sub-images and image sequence information corresponding to each sub-image.

[0063] In an embodiment of the present invention, a model based on CNN network training can be used to perform information recognition on the segmented sub-images, and an initial feature vector can be constructed based on the image sequence information corresponding to each identified sub-image and each sub-image to obtain an initial vector space.

[0064] Step S103: injecting preset weight information into the initial feature vector to obtain a weighted feature vector.

[0065] In an embodiment of the present invention, the preset weight information (for example, object and action weight information, such as "a person holding a dog" has a higher weight than "a dog holding a person") provided by the grammar check evaluation module can be received by the image information encoding module, and the preset weight information is injected into the initial feature vector (that is, the specific target subsequence of each image sequence information in the initial vector space) to obtain a weight feature vector (the weight information is injected into the initial vector space to obtain a weight vector space).

[0066] Optionally, after injecting preset weight information into the initial feature vector to obtain the weight feature vector, it also includes: when the image to be processed is an intermediate image in a continuous image set, receiving the encoding information of the previous image, wherein the intermediate image is any image in the continuous image set except the start image, and the previous image is an image at a time before the time of the image to be processed in the time series of the continuous image set; injecting the encoding information into the weight feature vector to obtain the target weight feature vector.

[0067] In an embodiment of the present invention, when the image to be processed is an intermediate image in a continuous image set (the intermediate image is any image in the continuous image set except the start image, and the previous image is an image at a previous time relative to the time of the image to be processed in the time sequence of the continuous image set, that is, the image to be processed is a video or an image in a series of images), the grammar check evaluation module can receive the encoding information of the previous image and transmit the encoding information to the information encoding processing module, so that the information encoding processing module can use the encoding information of the previous image as a reference, inject the encoding information into the weight feature vector, and obtain the target weight feature vector (the target weight feature vector includes the encoding information of this image, and can pass the encoding information of this image to the description information generation process of the next image). In this way, the extraction and inheritance of context information in streaming image processing (that is, in the processing of a video or a series of images) can be realized, and the role of partial grammar regulation reference can be provided.

[0068] Optionally, after injecting the encoded information into the weight feature vector to obtain the target weight feature vector, it also includes: based on a preset dimensionality reduction strategy, reducing the dimension of the target weight feature vector to obtain a reduced dimensionality feature vector; judging whether there is a similar classification relationship between the reduced dimensionality feature vectors to obtain a judgment result, wherein the similar classification relationship refers to the existence of image description information with equal weight values; when the judgment result indicates that the reduced dimensionality feature vectors have a similar classification relationship, the image description information in the previous image is used as the image description information in the image to be processed.

[0069] In an embodiment of the present invention, after injecting the encoded information into the weighted feature vector, the information encoding processing module may perform dimensionality reduction on the target weighted feature vector based on a preset dimensionality reduction strategy (e.g., PCA (Principal Component Analysis) dimensionality reduction) to obtain a reduced dimensionality feature vector. If, after dimensionality reduction in the high-dimensional space, a similar classification relationship is found, the image description information in the image to be processed is determined to be similar to the previous image description information, and the above grammatical encoding structure is referenced (i.e., a determination is made as to whether the reduced dimensionality feature vectors have a similar classification relationship, and a determination result is obtained. The similar classification relationship refers to the existence of image description information with equal weight values. If the determination result indicates that the reduced dimensionality feature vectors have a similar classification relationship, the image description information in the previous image is used as the image description information in the image to be processed). For example, when the image description is "boy picks up girl," but "girl picks up boy" also has a similar weight, but it is known through inherited information that it is "boy picks up girl," the current image is biased towards referencing the encoding result of "boy picks up girl" in the weight of the picking-up action.

[0070] Step S104 : decoding the weighted feature vector to generate a candidate information sequence, and parsing the candidate information sequence to obtain a plurality of image description information, wherein each image description information corresponds to an information evaluation score.

[0071] In an embodiment of the present invention, the weight feature vector can be identified and decoded based on the BiLSTM network model of the RNN, and a corresponding set of candidate information sequences can be generated. Afterwards, the BiLSTM model can be used to parse the candidate information sequence to generate multiple image description information, and output them to the grammar check evaluation module for evaluation, so that each image description information can correspond to an information evaluation score.

[0072] Optionally, after parsing the candidate information sequence to obtain multiple image description information, it also includes: based on a preset evaluation strategy, evaluating the multiple image description information to obtain an information evaluation score corresponding to each image description information; sorting the information evaluation scores to obtain a sorting result; based on the sorting result, selecting all information evaluation scores located before a preset number of bits as target evaluation scores; when the image to be processed is not the last image in a continuous image set, passing the encoded information of the image description information indicated by the target evaluation score into the image description information generation process of the next image.

[0073] In an embodiment of the present invention, a preset evaluation strategy (such as BLEU evaluation) can be used to evaluate multiple image description information to obtain an information evaluation score corresponding to each image description information. Afterwards, the information evaluation scores are sorted, and all information evaluation scores located before a preset number of digits (for example, the top three with the highest scores) are selected as target evaluation scores. In the case that the image to be processed is not the last image in a continuous image set, the encoding information of the image description information indicated by the target evaluation score is passed into the image description information generation process of the next image as a context reference.

[0074] Step S105 : selecting the image description information represented by the highest evaluation score as the target image description information of the image to be processed.

[0075] In an embodiment of the present invention, the image description information represented by the highest evaluation score may be used as the final image description information of the image to be processed and displayed.

[0076] In an embodiment of the present invention, a method for generating image description information with grammar checking and evaluation is provided, which can solve the problems of sentiment analysis ambiguity, semantic confusion, and omission caused by the lack of grammar checking and grammar reference in the current generation method, although it can form a text description based on the information contained in the picture. In addition, it can make the generated image description information have higher quality and be more in line with the grammatical rules of natural language, and can also reduce the number of manual revisions, thereby improving the efficiency and accuracy of image description information generation. Example 2

[0077] Figure 2 is a schematic diagram of an optional system for generating image description information according to an embodiment of the present invention. Figure 2 As shown, the generation system may include: an image information parsing module 1, an image information encoding module 2, a grammar checking and evaluation module 3, an information encoding processing module 4, and an image description generation module 5, wherein:

[0078] Image information parsing module 1 is used to receive an image to be processed, segment the image to be processed, obtain multiple segmented sub-images, construct an initial feature vector based on the sub-images and the image sequence information corresponding to each sub-image, and transmit the initial feature vector to the image information encoding module;

[0079] The image information encoding module 2 is connected to the image information parsing module, and is used to receive the initial feature vector, inject preset weight information into the initial feature vector, obtain the weight feature vector, and transmit the weight feature vector to the information encoding processing module;

[0080] The grammar check evaluation module 3 is connected to the image information encoding module and the image description generation module, and is used to transmit preset weight information to the image information encoding module, receive the image description information transmitted by the image description generation module, evaluate the image description information, and return the obtained information evaluation score to the image description generation module;

[0081] The information encoding processing module 4 is connected to the image information encoding module, and is used to receive the weight feature vector, decode the weight feature vector, generate a candidate information sequence, and transmit the candidate information sequence to the image description generation module;

[0082] The image description generation module 5 is connected to the information encoding processing module, and is used to receive the candidate information sequence and parse the candidate information sequence to obtain multiple image description information, and select the image description information represented by the highest evaluation score as the target image description information of the image to be processed, wherein each image description information corresponds to an information evaluation score.

[0083] In the embodiment of the present invention, the image information parsing module 1 is connected to the image information encoding module 2; the image information encoding module 2 is connected to the grammar check and evaluation module 3 and the information encoding processing module 4; the grammar check and evaluation module 3 is connected to the information encoding processing module 4, the image information encoding module 2, and the image description generation module 5; and the information encoding processing module 4 is connected to the image description generation module 5. The specific functions of each module are as follows:

[0084] The image information parsing module 1 is the encoder part, which can be composed of a CNN network. It can be used to parse the input image and the training image, and segment the image according to image information such as color, boundary and grayscale, and finally pass the analyzed and processed image to the image information encoding module 2.

[0085] The image information encoding module 2 is the encoder part, which can be composed of a CNN network and can be used to encode the image according to the parsed segmentation information and form a feature vector containing image information.

[0086] The grammar check evaluation module 3 can be used to inject a predefined weighted grammar information vector (i.e., weight information, for example, the action of "person riding a horse" has a high weight, but the action of "horse riding a person" has a low weight, and grammar information is used to reduce semantic logic errors) into the vector space generated by the image information encoding module 2, and receive the encoding information of the previous image (if there is a previous image), and inject it into the information encoding processing module as a reference. At the same time, the image description information generated by the image description generation module will be evaluated and an evaluation score will be given. The encoding information of the previous preset names with the highest scores (e.g., the top three) will be transmitted to the evaluation of the next image (if there is a next image) as a reference.

[0087] The information encoding processing module 4 can be based on the BiLSTM composition of the RNN network model (which is the Decoder decoder part), which can be used to decode the feature vector encoded by the image information encoding module 2 and generate the corresponding candidate information sequence. At the same time, it receives the encoding information of the previous image from the grammar check evaluation module 3 as a reference for generating the candidate information sequence, and finally passes the generated candidate information sequence to the image description generation module 5.

[0088] The image description generation module 5 can be based on the BiLSTM composition of the RNN network model, which can be used to parse the candidate information sequence into image description information, and pass the image description information back to the grammar check evaluation module 3 for evaluation, and generate a corresponding evaluation score, and display the image description information represented by the highest evaluation score as the final image description.

[0089] Figure 3 FIG. 1 is a schematic diagram of an optional structure of an image information analysis module 1 according to an embodiment of the present invention. Figure 3 As shown, the image information analysis module 1 includes: an image frame processing unit 11, an image frame segmentation unit 12, and an image information recognition unit 13, wherein:

[0090] The image processing unit 11 may extract image information such as preset cropping information, grayscale information, and color information from the input image, so that the input image meets the recognition requirements of the model to be used.

[0091] The image segmentation unit 12 may identify target information (including objects, scenes, and other information) of the processed input image and segment the objects, scenes, and other information in the image.

[0092] The image information recognition unit 13 can use a model based on CNN network training to perform information recognition on the segmented image, and deliver the recognized image sequence information to the image information encoding module 2.

[0093] Figure 4 FIG. 2 is a schematic diagram of an optional structure of an image information encoding module 2 according to an embodiment of the present invention. Figure 4 As shown, the image information encoding module 2 includes: an image information receiving unit 21, a weight information processing unit 22, and a model encoding unit 23, wherein:

[0094] The image information receiving unit 21 may receive the recognized image sequence information provided by the image information analyzing module 1 .

[0095] The weight information processing unit 22 can receive the object and action weight information provided by the grammar check evaluation module 3 (for example, the weight of "a person holding a dog" is higher than the weight of "a dog holding a person"), and inject the weight information into the specific target subsequence of the image sequence information.

[0096] The model encoding unit 23 can encode the weighted image information after the CNN model is recognized, that is, the image can be encoded into a weight feature vector through the action, position weight and image sequence information.

[0097] Optionally, the grammar check evaluation module includes: a weight information processing unit for receiving preset weight information, wherein the preset weight information includes at least: action weight information, position weight information, and master-slave weight information; an inheritance information receiving unit for receiving the encoding information of the previous image when the image to be processed is an intermediate image in a continuous image set, and injecting the encoding information into the weight feature vector, wherein the intermediate image is any image in the continuous image set except the start image, and the previous image is an image at a previous time relative to the time of the image to be processed in the time series of the continuous image set; an image description evaluation unit for evaluating the image description information based on a preset evaluation strategy to obtain an information evaluation score corresponding to each image description information.

[0098] In this embodiment, Figure 5 FIG. 3 is a schematic diagram of an optional structure of a grammar check evaluation module 3 according to an embodiment of the present invention. Figure 5 As shown, the grammar check evaluation module 3 includes: a weight information processing unit 31, an inheritance information receiving unit 32, and an image description evaluation unit 33, wherein:

[0099] The weight information processing unit 31 can be used to receive predefined weight information, which includes weight information such as action weight, position weight, and primary and secondary weight. The weight information can be generated according to the rule that the more common the action, position, primary and secondary are in real life, the higher the weight, and injected into the image sequence information.

[0100] The inherited information receiving unit 32 can be used to receive the coding information of the previous image (if there is a previous image) and inject it into the information coding processing module 4 to provide a context reference for the information coding of the current image. The current coding information can then be inherited into the next information coding (if there is a next image) processing as a context reference, thereby realizing the extraction and inheritance of context information in streaming image processing and providing some grammatical constraint references.

[0101] The image description evaluation unit 33 can evaluate the generated image description information (such as BLEU evaluation), and can use BLEU-4 (an evaluation method) for evaluation. The evaluation reference library supports manual import, and retains the results of the best image descriptions generated in the past for self-accumulation and iteration, and generates an evaluation score and returns it to the image description generation module 5.

[0102] Figure 6 Schematic diagram of an optional information encoding processing module 4 structure according to an embodiment of the present invention. Figure 6 As shown, the information encoding processing module 4 includes: an inheritance information receiving unit 41, an image information receiving unit 42, and an image information encoding unit 43, wherein:

[0103] The inherited information receiving unit 41 can receive the encoding information of the previous image (if any) sent by the grammar check and evaluation module 3, and inject the encoding information into the image feature vector space. After dimensionality reduction in the high-dimensional space (such as using PCA dimensionality reduction), if a similar classification relationship is found, it is determined to be similar to the previous image description result and reference is made to the grammatical encoding structure of the previous sentence. For example, in a scenario that the weight information processing unit 31 cannot distinguish: the image description is "a boy picked up a girl", but "a girl picked up a boy" also has a similar weight. However, through inherited information, it is known that it is "a boy picked up a girl". In this case, the weight of the picking-up action in this image will be biased towards the encoding result of "a boy picked up a girl".

[0104] The image information receiving unit 42 may receive a weighted feature vector including image features.

[0105] The image information encoding unit 43 may use a BiLSTM network model based on RNN to identify and decode the weight feature vector and generate a corresponding set of candidate information sequences.

[0106] Figure 7 FIG. 5 is a schematic diagram of an optional structure of an image description generating module 5 according to an embodiment of the present invention. Figure 7 As shown, the image description generating module 5 includes: an information sequence receiving unit 51, an image description generating unit 52, and an image description displaying unit 53, wherein:

[0107] The information sequence receiving unit 51 may receive the candidate information sequence output by the information encoding processing module 4 .

[0108] The image description generating unit 52 may use a BiLSTM model to generate image description information for the candidate information sequence and output the information to the grammar checking and evaluating module 3 for evaluation.

[0109] The image description display unit 53 may display the image description generation information having the highest evaluation score.

[0110] Figure 8 FIG. 1 is a schematic diagram of an optional image description generation process according to an embodiment of the present invention. Figure 8 As shown, the process includes the following:

[0111] Step S801: Input the image into the image information analysis module 1.

[0112] Step S802 : outputting the image analysis information after segmentation and recognition, and inputting the image analysis information into the image information encoding module 2 .

[0113] Step S803: The grammar checking and evaluation module 3 receives the inherited coding information.

[0114] Step S804 : the syntax check evaluation module 3 outputs weight information to the image information encoding module 2 .

[0115] Step S805: The image information encoding module 2 outputs weighted encoding information.

[0116] Step S806: The grammar check evaluation module 3 outputs the inherited coding information to the information coding processing module 4.

[0117] Step S807: The grammar check evaluation module 3 outputs the current encoding information to the evaluation result.

[0118] Step S808: the information encoding processing module 4 outputs image description sequence information.

[0119] Step S809 : The grammar check evaluation module 3 outputs the evaluation result to the image description generation module 5 .

[0120] Step S810: The image description generation module 5 outputs the image description information with the highest score as the final result and displays it.

[0121] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for generating image description information.

[0122] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned method for generating image description information.

[0123] Figure 9 FIG is a hardware structure block diagram of an electronic device (or mobile device) for a method for generating image description information according to an embodiment of the present invention. Figure 9 As shown, the electronic device may include one or more processors 902 (illustrated as 902a, 902b, ..., 902n in the figure) (the processor 902 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), and a memory 904 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that Figure 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 9 More or fewer components than shown, or with Figure 9 Different configurations shown.

[0124] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0125] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0127] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0128] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0129] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk.

[0130] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for generating image description information, characterized in that: include: receiving an image to be processed, and segmenting the image to be processed to obtain a plurality of segmented sub-images; constructing an initial feature vector based on the sub-images and image sequence information corresponding to each of the sub-images; Injecting preset weight information into the initial feature vector to obtain a weighted feature vector; In a case where the image to be processed is an intermediate image in a set of continuous images, receiving encoding information of a previous image, wherein the intermediate image is any image in the set of continuous images except the start image, and the previous image is an image at a time before the time of the image to be processed in the time sequence of the set of continuous images; injecting the encoding information into the weight feature vector to obtain a target weight feature vector; Decoding the weight feature vector to generate a candidate information sequence, and parsing the candidate information sequence to obtain a plurality of image description information, wherein each image description information corresponds to an information evaluation score; Selecting the image description information represented by the highest evaluation score as the target image description information of the image to be processed; Wherein, after injecting the coding information into the weight feature vector to obtain the target weight feature vector, the method further includes: Based on a preset dimensionality reduction strategy, the target weight feature vector is reduced in dimension to obtain a reduced-dimensional feature vector; Determining whether the reduced-dimensionality feature vectors have a similar classification relationship, and obtaining a determination result, wherein the similar classification relationship refers to the existence of image description information with equal weight values; When the judgment result indicates that the reduced-dimensionality feature vectors have a similar classification relationship, the image description information in the previous image is used as the image description information in the image to be processed.

2. The generation method according to claim 1, characterized in that After receiving the image to be processed, it also includes: Acquire preset image information, wherein the preset image information includes at least one of the following: color, boundary, and grayscale; Based on the preset image information, the image to be processed is preprocessed.

3. The generation method according to claim 1, characterized in that The step of segmenting the image to be processed to obtain a plurality of segmented sub-images includes: Identify the image to be processed to obtain target information, wherein the target information includes at least one of the following: an object, a scene; Based on the target information, the image to be processed is segmented to obtain a plurality of segmented sub-images.

4. The generation method according to claim 1, characterized in that After parsing the candidate information sequence to obtain a plurality of image description information, the method further includes: Based on a preset evaluation strategy, the plurality of image description information are evaluated to obtain an information evaluation score corresponding to each piece of the image description information; Sorting the information evaluation scores to obtain a ranking result; Based on the sorting result, all the information evaluation scores before the preset number of digits are selected as target evaluation scores; In a case where the image to be processed is not the last image in the continuous image set, the encoding information of the image description information indicated by the target evaluation score is passed into the image description information generation process of the next image.

5. A system for generating image description information, characterized in that: include: An image information parsing module is configured to receive an image to be processed, segment the image to be processed to obtain a plurality of segmented sub-images, construct an initial feature vector based on the sub-images and image sequence information corresponding to each sub-image, and transmit the initial feature vector to an image information encoding module; an image information encoding module, connected to the image information parsing module, configured to receive the initial feature vector, inject preset weight information into the initial feature vector to obtain a weighted feature vector, and transmit the weighted feature vector to the information encoding processing module; a grammar check evaluation module, connected to the image information encoding module and the image description generation module, configured to transmit the preset weight information to the image information encoding module, receive the image description information transmitted by the image description generation module, evaluate the image description information, and return the obtained information evaluation score to the image description generation module; an information encoding processing module, connected to the image information encoding module, configured to receive the weight feature vector, decode the weight feature vector, generate a candidate information sequence, and transmit the candidate information sequence to the image description generation module; an image description generation module connected to the information encoding processing module, configured to receive the candidate information sequence, parse the candidate information sequence, obtain multiple image description information, and select the image description information represented by the highest evaluation score as the target image description information of the image to be processed, wherein each image description information corresponds to an information evaluation score; an inheritance information receiving unit, configured to receive encoding information of a previous image when the image to be processed is an intermediate image in a set of continuous images, and inject the encoding information into the weight feature vector to obtain a target weight feature vector, wherein the intermediate image is any image in the set of continuous images except the start image, and the previous image is an image at a time before the time of the image to be processed in the time sequence of the set of continuous images; A similarity judgment module is used to reduce the dimension of the target weight feature vector based on a preset dimensionality reduction strategy to obtain a reduced dimension feature vector; judge whether the reduced dimension feature vector has a similar classification relationship to obtain a judgment result, wherein the similar classification relationship refers to the existence of image description information with equal weight values; when the judgment result indicates that the reduced dimension feature vector has a similar classification relationship, the image description information in the previous image is used as the image description information in the image to be processed.

6. The generation system according to claim 5, characterized in that The grammar check evaluation module includes: A weight information processing unit, configured to receive the preset weight information, wherein the preset weight information includes at least: action weight information, position weight information, and master-slave weight information; The image description evaluation unit is used to evaluate the image description information based on a preset evaluation strategy to obtain an information evaluation score corresponding to each piece of image description information.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for generating image description information according to any one of claims 1 to 4.

8. An electronic device, characterized in that: The device comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating image description information as described in any one of claims 1 to 4.

Images