Text generation method and device based on artificial intelligence, equipment and storage medium

Abstract

The invention provides a text generation method and device based on artificial intelligence, equipment and a storage medium, and belongs to the field of artificial intelligence, and the method comprises the steps: obtaining a target video of a to-be-extracted semantic text; obtaining a text generation model; performing feature extraction on each image frame in the target video through a feature extraction module to obtain a first image feature vector; inputting the first image feature vector into an attractor guide module to update an attractor token to obtain attractor state information; inputting the attractor state information and the first image feature vector into a feature fusion module for feature fusion to obtain a second image feature vector; and decoding the second image feature vector and the attractor state information through a text output module, and outputting a target text. The efficiency and the accuracy of generating the text data by the video are improved. The method can be applied to the technical fields of finance and medical treatment, and the accuracy of converting financial product videos and medical related videos into semantic texts is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based text generation method, apparatus, device, and storage medium. Background Art

[0002] With the continuous development of information and communications technologies, videos are becoming increasingly popular and applicable in a wide range of scenarios. For example, in the financial industry, insurance companies use short videos to introduce insurance products to users, while banks and financial institutions use short videos to introduce financial products to users. Another example is the healthcare industry, where videos can be used to explain how to use medical devices to medical staff, or to explain the functions, precautions, and dosages of medications to patients. While videos have greatly facilitated people's lives, they can also have drawbacks. For example, watching videos about financial products can take time, and it can be difficult to quickly find important information about the products, making them less intuitive than text-based descriptions. For example, deaf and mute patients who watch videos explaining the functions, precautions, and dosages of medications cannot hear the audio, causing inconvenience.

[0003] When videos are inconvenient for users, it's necessary to extract semantic information from the video and output it as text data so that users can more intuitively access the video information. However, currently, outputting video data as coherent text data is ineffective, resulting in significant errors in understanding the video information.

[0004] Therefore, how to accurately extract semantics from video data and output text data is a problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of this application is to provide an artificial intelligence-based text generation method, device, equipment and storage medium, aiming to improve the efficiency and accuracy of video semantic generation of text data.

[0006] In a first aspect, the present application provides a method for generating text based on artificial intelligence, the method comprising the following steps:

[0007] Obtaining a target video for semantic text extraction, wherein the target video includes multiple image frames;

[0008] Obtaining a text generation model, the text generation model comprising a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module, the text generation model being obtained by pre-training a neural network model based on a plurality of training samples, the training samples comprising a sample video and annotated target text;

[0009] Performing feature extraction on each image frame in the target video by the feature extraction module to obtain a first image feature vector;

[0010] Inputting the first image feature vector into the attractor guidance module to update the attractor token to obtain attractor state information;

[0011] Inputting the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector;

[0012] The second image feature vector and the attractor state information are decoded by the text output module to output the target text.

[0013] In a second aspect, the present application further provides a text generation device, which includes an acquisition module and a generation module, wherein:

[0014] The acquisition module is used to acquire a target video for extracting semantic text, wherein the target video includes multiple image frames;

[0015] The acquisition module is further used to acquire a text generation model, which includes a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module. The text generation model is obtained by pre-training a neural network model based on multiple training samples, and the training samples include sample videos and annotated target texts;

[0016] The generating module is configured to extract features from each image frame in the target video through the feature extraction module to obtain a first image feature vector;

[0017] The generating module is further configured to input the first image feature vector into the attractor guiding module to update the attractor token and obtain attractor state information;

[0018] The generating module is further configured to input the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector;

[0019] The generation module is further configured to decode the second image feature vector and the attractor state information through the text output module, and output a target text.

[0020] In a third aspect, the present application also provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the above-mentioned artificial intelligence-based text generation method are implemented.

[0021] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-mentioned artificial intelligence-based text generation method are implemented.

[0022] The present application provides a text generation method, apparatus, device and storage medium based on artificial intelligence. The present application obtains a target video for semantic text extraction, the target video including multiple image frames; obtains a text generation model, the text generation model including a feature extraction module, an attractor guidance module, a feature fusion module and a text output module, the text generation model is obtained by pre-training a neural network model based on multiple training samples, the training samples including sample videos and annotated target texts; the feature extraction module extracts features from each image frame in the target video to obtain a first image feature vector; the first image feature vector is input into the attractor guidance module to update the attractor token to obtain attractor state information; the attractor state information and the first image feature vector are input into the feature fusion module to perform feature fusion to obtain a second image feature vector; the second image feature vector and the attractor state information are decoded and processed by the text output module to output the target text. In the present application, the target video is subjected to video semantic extraction by each module in the text generation model, and the target text corresponding to the target video can be accurately generated. The attractor guidance module in the text generation model performs semantic recognition, which can effectively improve the accuracy of the output target text, thereby greatly improving the efficiency and accuracy of video semantic generation text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 A flowchart of an artificial intelligence-based text generation method provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of another method for generating text based on artificial intelligence provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a text generation model provided in an embodiment of the present application;

[0027] Figure 4 A schematic block diagram of a text generation device provided in an embodiment of the present application;

[0028] Figure 5 A schematic block diagram of another text generation device provided in an embodiment of the present application;

[0029] Figure 6 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0030] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0033] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0034] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0035] With the continuous development of information and communications technologies, videos are becoming increasingly popular and applicable in a wide range of scenarios. For example, in the financial industry, insurance companies use short videos to introduce insurance products to users, while banks and financial institutions use short videos to introduce financial products to users. Another example is the healthcare industry, where videos can be used to explain how to use medical devices to medical staff, or to explain the functions, precautions, and dosages of medications to patients. While videos have greatly facilitated people's lives, they can also have drawbacks. For example, watching videos about financial products can take time, and it can be difficult to quickly find important information about the products, making them less intuitive than text-based descriptions. For example, deaf and mute patients who watch videos explaining the functions, precautions, and dosages of medications cannot hear the audio, causing inconvenience.

[0036] When videos are inconvenient for users, it's necessary to extract semantic information from the video and output it as text data so that users can more intuitively access the video information. However, currently, outputting video data as coherent text data is ineffective, resulting in significant errors in understanding the video information.

[0037] To solve the above problems, the embodiments of the present application provide a method, apparatus, device and storage medium for text generation based on artificial intelligence. The method for text generation based on artificial intelligence includes: obtaining a target video for semantic text extraction, wherein the target video includes multiple image frames; obtaining a text generation model, wherein the text generation model includes a feature extraction module, an attractor guidance module, a feature fusion module and a text output module, wherein the text generation model is obtained by pre-training a neural network model based on multiple training samples, wherein the training samples include sample videos and annotated target texts; extracting features from each image frame in the target video through the feature extraction module to obtain a first image feature vector; inputting the first image feature vector into the attractor guidance module to update the attractor token to obtain attractor state information; inputting the attractor state information and the first image feature vector into the feature fusion module to perform feature fusion to obtain a second image feature vector; decoding the second image feature vector and the attractor state information through the text output module to output the target text.

[0038] Among them, the artificial intelligence-based text generation method can be applied to computer devices, which can be electronic devices such as mobile phones, tablet computers, laptops, desktop computers, personal digital assistants and wearable devices.

[0039] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0040] Please refer to Figure 1 , Figure 1 A flowchart of an artificial intelligence-based text generation method provided in an embodiment of the present application.

[0041] like Figure 1 As shown, the text generation method based on artificial intelligence includes steps S101 to S106.

[0042] Step S101: Obtain a target video from which semantic text is to be extracted, wherein the target video includes multiple image frames.

[0043] The target video includes, but is not limited to, a video introducing a financial product, a video on how to use a medical device, a video introducing the functions of a drug, a sports video, etc. The target video includes multiple image frames. The number of image frames included in the target video can be set according to actual conditions and is not specifically limited in this embodiment of the application. For example, the target video includes 20 image frames.

[0044] In some embodiments, a video of the semantic text to be extracted is obtained to obtain a target video. By obtaining the video of the semantic text to be extracted, the target video can be accurately obtained.

[0045] Step S102: Acquire a text generation model.

[0046] Among them, the text generation model includes a feature extraction module, an attractor guidance module, a feature fusion module and a text output module. The text generation model is obtained by pre-training a neural network model based on multiple training samples, and the training samples include sample videos and annotated target texts.

[0047] In some embodiments, as Figure 2 As shown, the artificial intelligence-based text generation method also includes steps S201 to S205.

[0048] Step S201: Acquire a sample data set, where the sample data set includes a plurality of sample data, and the sample data includes a sample video and annotated target text.

[0049] The sample data set includes a plurality of sample data, and the sample data includes sample videos and annotated target texts.

[0050] In some embodiments, historical data is obtained, the historical data comprising video data and target text corresponding to the video data, wherein the target text is text extracted and verified from the video data. The video data in the historical data is used as a sample video, and the target text corresponding to the video data is annotated to obtain the annotated target text. The aforementioned steps are repeated in a loop to obtain a sample dataset.

[0051] Step S202: Obtain a preset text generation model, and select a sample data from the sample data set as target sample data.

[0052] Among them, the preset text generation model is a neural network model, which includes but is not limited to a convolutional neural network model, a recurrent neural network model, a recurrent convolutional neural network model, and the like.

[0053] It should be noted that if Figure 3 As shown, the preset text generation model includes a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module. The feature extraction module is used to extract features from each image frame in the video to obtain an image feature vector; the attractor guidance module is used to output the attractor state to adjust the features of the image feature vector according to the attractor state; the feature fusion module is used to output the fused image feature vector; and the text output module is used to decode the feature vector to output text.

[0054] Among them, the specific models selected by the feature extraction module, attractor guidance module, feature fusion module and text output module can be set according to actual conditions, and the embodiments of the present application do not specifically limit this. For example, the feature extraction module includes SwinTransformer-V2 and 3D ConvNeXt, and each image frame in the video is divided by SwinTransformer-V2, and features are extracted by 3D ConvNeXt. The attractor guidance module includes a long short-term memory network and an attractor token. The feature fusion module includes a Transformer encoder. The text output module includes a T5-base decoder.

[0055] In some embodiments, a sample data is randomly selected from the sample data set as the target sample data. The target sample includes a sample video and annotated target text. By selecting a sample data from the sample data set, the target sample data can be accurately obtained.

[0056] Step S203: inputting the sample video of the sample data in the target sample data into the preset text generation model to generate a predicted target text.

[0057] In some embodiments, a feature extraction module extracts features from each image frame in a sample video to obtain a predicted first image feature vector. This predicted first image feature vector is then fed into an attractor guidance module to update the attractor token, thereby obtaining predicted attractor state information. This predicted attractor state information and the predicted first image feature vector are then fed into a feature fusion module for feature fusion to obtain a predicted second image feature vector. This predicted second image feature vector and the predicted attractor state information are then decoded and processed by a text output module to output a predicted target text. By feeding the sample video into a preset text generation model, the predicted target text can be accurately generated.

[0058] In some embodiments, the predicted first image feature vector is input into an attractor guidance module to update the attractor token, and the method for obtaining the predicted attractor state information can be: the attractor guidance module includes a token initialization layer and a token update layer; the predicted first image feature vector is input into the token initialization layer to initialize the token to obtain predicted initial attractor state information, wherein the predicted initial attractor state information includes multiple predicted initial attractor coordinates; the predicted first image feature vector and the predicted initial attractor state information are input into the predicted token update layer to update the attractor coordinates to obtain the predicted attractor state information. By inputting the predicted first image feature vector into the attractor guidance module to update the attractor token, the predicted attractor state information can be accurately obtained.

[0059] In some embodiments, the predicted attractor state information and the predicted first image feature vector are input into a feature fusion module for feature fusion, and the method for obtaining the predicted second image feature vector can be: constructing a query matrix for the predicted first image feature vector through the feature fusion module to obtain a query matrix of the predicted target video; constructing a key matrix for the predicted attractor state information through the feature fusion module to obtain a predicted key matrix; and performing feature fusion on the predicted query matrix and the predicted key matrix through the feature fusion module to obtain a predicted second image feature vector.

[0060] In some embodiments, the predicted second image feature vector and the predicted attractor state information are decoded through a text output module, and the predicted target text is outputted in the following manner: the text output module includes a timing output layer and a decoder layer; the predicted second image feature vector is subjected to timing feature extraction through the timing output layer to obtain a predicted timing feature vector; the predicted timing feature vector, the predicted second image feature vector and the predicted attractor state information are decoded through the decoder layer to output the predicted target text.

[0061] In some embodiments, a decoder layer decodes the predicted time series feature vector, the predicted second image feature vector, and the predicted attractor state information, and outputs the predicted target text by: decoding the predicted time series feature vector, the predicted second image feature vector, and the predicted attractor state information to obtain predicted video description text information; and performing text refinement extraction on the predicted video description text information to obtain the predicted target text. By performing text refinement extraction on the predicted video description text information, the predicted target text can be accurately obtained.

[0062] Step S204: Determine whether the preset text generation model has converged based on the predicted target text and the annotated target text.

[0063] Based on the predicted target text and the annotated target text, a loss value of a preset text generation model is determined; if the loss value of the preset text generation model is less than or equal to the preset loss value, it is determined that the preset text generation model has converged; if the loss value of the preset text generation model is greater than the preset loss value, it is determined that the preset text generation model has not converged. The preset loss value can be set according to actual conditions and is not specifically limited in the embodiments of the present application. For example, the preset loss value can be set to 0.02.

[0064] In some embodiments, the loss value of a preset text generation model can be determined based on the predicted target text and the annotated target text by: calculating the similarity between the predicted target text and the annotated target text to obtain a first similarity; obtaining a historical similarity, which is the average of the first similarities of each sample video that has completed training, and performing an average calculation on the first similarity and the historical similarity to obtain a target similarity; subtracting the target similarity from unit one, and the obtained value is determined as the loss value.

[0065] In some embodiments, the loss value of a preset text generation model can be determined based on the predicted target text and the annotated target text by obtaining a first similarity, subtracting the target similarity from 1, and determining the resulting value as the first loss value; obtaining a second loss value, and averaging the first and second loss values ​​to determine the resulting value as the target loss value. Averaging multiple loss values ​​can improve the accuracy of the loss value.

[0066] In some embodiments, the second loss value may be obtained by obtaining a preset loss value formula, which is: is the second loss value, K is the number of attractor coordinates, A is the attractor state information, and σ is the bandwidth constant. Based on the preset loss value formula, the predicted attractor state information and the number of attractor coordinates are calculated to accurately obtain the second loss value.

[0067] Step S205: If the preset text generation model has not converged, continue to execute the step of selecting a sample data from the sample data set as the target sample data until the preset text generation model converges.

[0068] If the loss value of the preset text generation model is greater than the preset loss value, it is determined that the preset text generation model has not converged; a sample data is selected from the sample data set as the target sample data in a loop; a sample video of the sample data in the target sample data is input into the preset text generation model to generate a predicted target text; based on the predicted target text and the annotated target text, it is determined whether the preset text generation model has converged, until the preset text generation model converges, thereby obtaining the convergence of the text generation model.

[0069] Step S103: extract features from each image frame in the target video using the feature extraction module to obtain a first image feature vector.

[0070] The target video is input into a feature extraction module to perform feature extraction on each image frame to obtain a first image feature vector, wherein the first image feature vector includes a feature vector of each image frame.

[0071] Exemplarily, feature extraction is performed on the image frame using SwinTransformer-V2 to obtain a grid feature vector for each image frame, and time-series compression is performed on the grid feature vector for each image frame using 3D ConvNeXt to obtain an image feature vector for each image frame.

[0072] Step S104: input the first image feature vector into the attractor guidance module to update the attractor token to obtain attractor state information.

[0073] Among them, the attractor guidance module includes a token initialization layer and a token update layer. The model types of the token initialization layer and the token update layer can be selected according to actual conditions. The embodiment of the present application does not make specific limitations on this. For example, the token initialization layer can be a long short-term memory network, and the token update layer can be an attractor token model.

[0074] In some embodiments, the first image feature vector is input into the token initialization layer for token initialization to obtain initial attractor state information, which includes multiple initial attractor coordinates; the first image feature vector and the initial attractor state information are input into the token update layer for attractor coordinate update to obtain attractor state information.

[0075] Step S105: Input the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector.

[0076] The feature fusion module constructs a query matrix for the first image feature vector to obtain the query matrix of the target video. The feature fusion module constructs a key matrix for the attractor state information to obtain the key matrix. The feature fusion module fuses the query matrix and the key matrix to obtain the second image feature vector. The feature fusion module fuses the attractor state information with the features of the first image feature vector to accurately obtain the second image feature vector.

[0077] For example, a preset feature fusion formula is obtained, which is: is the second image feature vector, v t is the first image feature vector, G t,k is the coupling weight between the video and the attractor, A k is the attractor state information; based on the preset feature fusion formula, the attractor state information and the first image feature vector are calculated to obtain the second image feature vector.

[0078] It should be noted that the coupling weight G between the video and the attractor t,k The calculation method can be: Get the preset weight calculation formula, the preset weight calculation formula is G t,k is the coupling weight between the video and the attractor, λ is the preset first coefficient, q t is the query matrix, W p is the alignment matrix of attractor state information, a k is the key matrix, and τ is the preset second coefficient. Based on the preset weight calculation formula, the preset first coefficient, the query matrix, the alignment matrix of the attractor state information, the key matrix, and the preset second coefficient are calculated to obtain the coupling weight between the video and the attractor. The preset first coefficient and the preset second coefficient can be set according to actual conditions and are not specifically limited in this embodiment of the present application. For example, the preset first coefficient can be set to 5, and the preset second coefficient can be set to 0.3.

[0079] Step S106: decoding the second image feature vector and the attractor state information through the text output module to output the target text.

[0080] The text output module includes a temporal output layer and a decoder layer. The temporal output layer may be a 3DConvNeXt, and the decoder layer may be a T5-base decoder.

[0081] In some embodiments, a temporal output layer extracts temporal features from the second image feature vector to obtain a temporal feature vector; a decoder layer decodes the temporal feature vector, the second image feature vector, and the attractor state information to output the target text. The text output module decodes the second image feature vector and the attractor state information to accurately obtain the target text.

[0082] In some embodiments, the decoder layer decodes the temporal feature vector, the second image feature vector, and the attractor state information to output the target text. The decoder layer decodes the temporal feature vector, the second image feature vector, and the attractor state information to obtain video description text information; and then performs text refinement extraction on the video description text information to obtain the target text. By performing text refinement extraction on the video description text information, a more accurate target text can be obtained, significantly improving the efficiency and accuracy of video semantic extraction.

[0083] Exemplarily, a financial product introduction video of the semantic text to be extracted is obtained, and a text generation model is obtained, the text generation model including a feature extraction module, an attractor guidance module, a feature fusion module and a text output module; the feature extraction module is used to perform feature extraction on each image frame in the financial product introduction video to obtain a first image feature vector; the first image feature vector is input into the attractor guidance module to update the attractor token to obtain attractor state information; the attractor state information and the first image feature vector are input into the feature fusion module to perform feature fusion to obtain a second image feature vector; the second image feature vector and the attractor state information are decoded and processed by the text output module to output the financial product introduction text, which includes the purchase amount of the financial product, the income and risk information of each time period.

[0084] Exemplarily, a video of the method of using a medical device with semantic text to be extracted is obtained, and a text generation model is obtained, the text generation model including a feature extraction module, an attractor guidance module, a feature fusion module and a text output module; the feature extraction module is used to perform feature extraction on each image frame in the video of the method of using the medical device to obtain a first image feature vector; the first image feature vector is input into the attractor guidance module to update the attractor token to obtain attractor state information; the attractor state information and the first image feature vector are input into the feature fusion module to perform feature fusion to obtain a second image feature vector; the second image feature vector and the attractor state information are decoded and processed by the text output module to output a text of the method of using the medical device, wherein the text of the method of using the medical device includes steps for using the medical device and precautions for use.

[0085] The above embodiment provides an artificial intelligence-based text generation method, which obtains a target video for semantic text extraction, wherein the target video includes multiple image frames; obtains a text generation model, wherein the text generation model includes a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module, wherein the text generation model is obtained by pre-training a neural network model based on multiple training samples, wherein the training samples include sample videos and annotated target texts; the feature extraction module extracts features from each image frame in the target video to obtain a first image feature vector; the first image feature vector is input into the attractor guidance module for attractor token update to obtain attractor state information; the attractor state information and the first image feature vector are input into the feature fusion module for feature fusion to obtain a second image feature vector; the second image feature vector and the attractor state information are decoded and processed by the text output module to output the target text. In this application, the target video is subjected to video semantic extraction by each module in the text generation model, and the target text corresponding to the target video can be accurately generated. The attractor guidance module in the text generation model performs semantic recognition, which can effectively improve the accuracy of the output target text, thereby greatly improving the efficiency and accuracy of video semantic generation text data.

[0086] See 4, Figure 4 A schematic block diagram of a text generation device provided in an embodiment of the present application.

[0087] like Figure 4 As shown, the text generation device 300 includes an acquisition module 310 and a generation module 320, wherein:

[0088] The acquisition module 310 is used to acquire a target video for semantic text extraction, wherein the target video includes multiple image frames;

[0089] The acquisition module 310 is further configured to acquire a text generation model, which includes a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module. The text generation model is obtained by pre-training a neural network model based on multiple training samples, wherein the training samples include sample videos and annotated target texts.

[0090] The generating module 320 is configured to extract features from each image frame in the target video using the feature extraction module to obtain a first image feature vector;

[0091] The generating module 320 is further configured to input the first image feature vector into the attractor guiding module to update the attractor token and obtain attractor state information;

[0092] The generating module 320 is further configured to input the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector;

[0093] The generating module 320 is further configured to decode the second image feature vector and the attractor state information through the text output module to output a target text.

[0094] In some embodiments, the generating module 320 is further configured to:

[0095] Inputting the first image feature vector into the token initialization layer for token initialization to obtain initial attractor state information, wherein the initial attractor state information includes a plurality of initial attractor coordinates;

[0096] The first image feature vector and the initial attractor state information are input into the token update layer to update the attractor coordinates to obtain the attractor state information.

[0097] In some embodiments, the generating module 320 is further configured to:

[0098] Constructing a query matrix for the first image feature vector using the feature fusion module to obtain a query matrix for the target video;

[0099] Constructing a key matrix for the attractor state information by the feature fusion module to obtain a key matrix;

[0100] The query matrix and the key matrix are subjected to feature fusion by the feature fusion module to obtain a second image feature vector.

[0101] In some embodiments, the generating module 320 is further configured to:

[0102] Performing time series feature extraction on the second image feature vector through the time series output layer to obtain a time series feature vector;

[0103] The time series feature vector, the second image feature vector and the attractor state information are decoded by the decoder layer to output the target text.

[0104] In some embodiments, the generating module 320 is further configured to:

[0105] Decoding the time series feature vector, the second image feature vector, and the attractor state information to obtain video description text information;

[0106] The video description text information is refined and extracted to obtain the target text.

[0107] See 5, Figure 5 A schematic block diagram of another text generation device provided in an embodiment of the present application.

[0108] like Figure 5 As shown, the text generation device 400 includes an acquisition module 410, a selection module 420, a generation module 430 and a determination module 440, wherein:

[0109] The acquisition module 410 is used to acquire a sample data set, where the sample data set includes a plurality of sample data, and the sample data includes a sample video and annotated target text;

[0110] The acquisition module 410 is further used to acquire a preset text generation model;

[0111] The selection module 420 is used to select a sample data from the sample data set as the target sample data;

[0112] The generating module 430 is configured to input a sample video of the sample data in the target sample data into the preset text generation model to generate a predicted target text;

[0113] The determination module 440 is configured to determine whether the preset text generation model has converged based on the predicted target text and the annotated target text;

[0114] The generation module 430 is configured to continue executing the step of selecting a sample data from the sample data set as target sample data if the preset text generation model has not converged, until the preset text generation model converges.

[0115] In some embodiments, the determining module 440 is further configured to:

[0116] Determining a loss value of the preset text generation model according to the predicted target text and the annotated target text;

[0117] If the loss value of the preset text generation model is less than or equal to the preset loss value, determining that the preset text generation model has converged;

[0118] If the loss value of the preset text generation model is greater than the preset loss value, it is determined that the preset text generation model has not converged.

[0119] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned text generation device can refer to the corresponding process in the aforementioned embodiment of the artificial intelligence-based text generation method, and will not be repeated here.

[0120] See also Figure 6, Figure 6 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0121] like Figure 6 As shown, the computer device 500 includes a processor 502 and a memory 503 connected via a system bus 501 , wherein the memory 503 may include a storage medium and an internal memory.

[0122] The storage medium may store a computer program. The computer program includes program instructions, which, when executed, may cause a processor to execute any one of the artificial intelligence-based text generation methods.

[0123] The processor 502 is used to provide computing and control capabilities and support the operation of the entire computer device.

[0124] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any text generation method based on artificial intelligence.

[0125] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0126] It should be understood that the processor 502 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0127] In one embodiment, the processor 502 is configured to execute a computer program stored in a memory to implement the following steps:

[0128] Obtaining a target video for semantic text extraction, wherein the target video includes multiple image frames;

[0129] Obtaining a text generation model, the text generation model comprising a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module, the text generation model being obtained by pre-training a neural network model based on a plurality of training samples, the training samples comprising a sample video and annotated target text;

[0130] Performing feature extraction on each image frame in the target video by the feature extraction module to obtain a first image feature vector;

[0131] Inputting the first image feature vector into the attractor guidance module to update the attractor token to obtain attractor state information;

[0132] Inputting the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector;

[0133] The second image feature vector and the attractor state information are decoded by the text output module to output the target text.

[0134] In one embodiment, the attractor guidance module includes a token initialization layer and a token update layer; when the processor 502 inputs the first image feature vector into the attractor guidance module to update the attractor token and obtains the attractor state information, it is configured to implement:

[0135] Inputting the first image feature vector into the token initialization layer for token initialization to obtain initial attractor state information, wherein the initial attractor state information includes a plurality of initial attractor coordinates;

[0136] The first image feature vector and the initial attractor state information are input into the token update layer to update the attractor coordinates to obtain the attractor state information.

[0137] In one embodiment, when inputting the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain the second image feature vector, the processor 502 is configured to implement:

[0138] Constructing a query matrix for the first image feature vector using the feature fusion module to obtain a query matrix for the target video;

[0139] Constructing a key matrix for the attractor state information by the feature fusion module to obtain a key matrix;

[0140] The query matrix and the key matrix are subjected to feature fusion by the feature fusion module to obtain a second image feature vector.

[0141] In one embodiment, the text output module includes a temporal output layer and a decoder layer; when the processor 502 decodes the second image feature vector and the attractor state information through the text output module and outputs the target text, it is configured to implement:

[0142] Performing time series feature extraction on the second image feature vector through the time series output layer to obtain a time series feature vector;

[0143] The time series feature vector, the second image feature vector and the attractor state information are decoded by the decoder layer to output the target text.

[0144] In one embodiment, when the processor 502 decodes the temporal feature vector, the second image feature vector, and the attractor state information through the decoder layer and outputs the target text, it is configured to implement:

[0145] Decoding the time series feature vector, the second image feature vector, and the attractor state information to obtain video description text information;

[0146] The video description text information is refined and extracted to obtain the target text.

[0147] In one embodiment, the processor 502 is further configured to implement:

[0148] Acquire a sample data set, where the sample data set includes a plurality of sample data, and the sample data includes a sample video and annotated target text;

[0149] Obtaining a preset text generation model, and selecting a sample data from the sample data set as target sample data;

[0150] Inputting a sample video of the sample data in the target sample data into the preset text generation model to generate a predicted target text;

[0151] Determining whether the preset text generation model converges based on the predicted target text and the annotated target text;

[0152] If the preset text generation model has not converged, the step of selecting a sample data from the sample data set as the target sample data is continued until the preset text generation model converges.

[0153] In one embodiment, when determining whether the preset text generation model has converged based on the predicted target text and the annotated target text, the processor 502 is configured to implement:

[0154] Determining a loss value of the preset text generation model according to the predicted target text and the annotated target text;

[0155] If the loss value of the preset text generation model is less than or equal to the preset loss value, determining that the preset text generation model has converged;

[0156] If the loss value of the preset text generation model is greater than the preset loss value, it is determined that the preset text generation model has not converged.

[0157] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the above description of the specific working process of the computer device can refer to the corresponding process in the aforementioned embodiment of the text generation method based on artificial intelligence, and will not be repeated here.

[0158] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the artificial intelligence-based text generation method of the present application.

[0159] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0160] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0161] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0162] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0163] It should also be understood that the term "and / or" used in this specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0164] The serial numbers of the embodiments of the present application are for descriptive purposes only and do not represent the merits of the embodiments. The above description is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A text generation method based on artificial intelligence, characterized in that: include: Obtaining a target video for semantic text extraction, wherein the target video includes multiple image frames; Obtaining a text generation model, the text generation model comprising a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module, the text generation model being obtained by pre-training a neural network model based on a plurality of training samples, the training samples comprising a sample video and annotated target text; Performing feature extraction on each image frame in the target video by the feature extraction module to obtain a first image feature vector; Inputting the first image feature vector into the attractor guidance module to update the attractor token to obtain attractor state information; Inputting the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector; The second image feature vector and the attractor state information are decoded by the text output module to output the target text.

2. The text generation method based on artificial intelligence according to claim 1, characterized in that: The attractor guidance module includes a token initialization layer and a token update layer; the first image feature vector is input into the attractor guidance module to update the attractor token to obtain attractor state information, including: Inputting the first image feature vector into the token initialization layer for token initialization to obtain initial attractor state information, wherein the initial attractor state information includes a plurality of initial attractor coordinates; The first image feature vector and the initial attractor state information are input into the token update layer to update the attractor coordinates to obtain the attractor state information.

3. The text generation method based on artificial intelligence according to claim 1, characterized in that: The step of inputting the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector includes: Constructing a query matrix for the first image feature vector using the feature fusion module to obtain a query matrix for the target video; Constructing a key matrix for the attractor state information by the feature fusion module to obtain a key matrix; The query matrix and the key matrix are subjected to feature fusion by the feature fusion module to obtain a second image feature vector.

4. The text generation method based on artificial intelligence according to claim 1, characterized in that: The text output module includes a time sequence output layer and a decoder layer; the text output module decodes the second image feature vector and the attractor state information to output the target text, including: Performing time series feature extraction on the second image feature vector through the time series output layer to obtain a time series feature vector; The time series feature vector, the second image feature vector and the attractor state information are decoded by the decoder layer to output the target text.

5. The text generation method based on artificial intelligence according to claim 4, characterized in that: The decoding process of the time series feature vector, the second image feature vector and the attractor state information by the decoder layer to output the target text includes: Decoding the time series feature vector, the second image feature vector, and the attractor state information to obtain video description text information; The video description text information is refined and extracted to obtain the target text.

6. The text generation method based on artificial intelligence according to claim 1, characterized in that: The method further comprises: Acquire a sample data set, where the sample data set includes a plurality of sample data, and the sample data includes a sample video and annotated target text; Obtaining a preset text generation model, and selecting a sample data from the sample data set as target sample data; Inputting a sample video of the sample data in the target sample data into the preset text generation model to generate a predicted target text; Determining whether the preset text generation model converges based on the predicted target text and the annotated target text; If the preset text generation model has not converged, the step of selecting a sample data from the sample data set as the target sample data is continued until the preset text generation model converges.

7. The text generation method based on artificial intelligence according to claim 6, characterized in that: The determining, based on the predicted target text and the annotated target text, whether the preset text generation model has converged includes: Determining a loss value of the preset text generation model according to the predicted target text and the annotated target text; If the loss value of the preset text generation model is less than or equal to the preset loss value, determining that the preset text generation model has converged; If the loss value of the preset text generation model is greater than the preset loss value, it is determined that the preset text generation model has not converged.

8. A text generation device, characterized in that: The text generation device includes an acquisition module and a generation module, wherein: The acquisition module is used to acquire a target video for extracting semantic text, wherein the target video includes multiple image frames; The acquisition module is further used to acquire a text generation model, which includes a feature extraction module, an attractor guidance module, a feature fusion module, and a text output module. The text generation model is obtained by pre-training a neural network model based on multiple training samples, and the training samples include sample videos and annotated target texts; The generating module is configured to extract features from each image frame in the target video through the feature extraction module to obtain a first image feature vector; The generating module is further configured to input the first image feature vector into the attractor guiding module to update the attractor token and obtain attractor state information; The generating module is further configured to input the attractor state information and the first image feature vector into the feature fusion module for feature fusion to obtain a second image feature vector; The generation module is further configured to decode the second image feature vector and the attractor state information through the text output module, and output a target text.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the artificial intelligence-based text generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the artificial intelligence-based text generation method according to any one of claims 1 to 7 are implemented.

Images