Text prediction method and device based on virtual scene and electronic device

Abstract

The present disclosure relates to the technical field of images, and specifically discloses a text prediction method and device based on a virtual scene and electronic equipment. The method comprises: obtaining initial text in the virtual scene, and determining text features corresponding to the initial text; obtaining a virtual scene image corresponding to the initial text in the virtual scene, and determining image features of the virtual scene image; performing interactive fusion processing on the text features of the initial text according to the image features of the virtual scene image, to obtain text fusion features of the initial text; and predicting subsequent text in the virtual scene according to the text fusion features of the initial text. Since the text fusion features in this way belong to multi-modal features, they can more comprehensively reflect the characteristics of the virtual scene, so that the prediction result of the subsequent text is more accurate and efficient.

Description

Technical Field

[0001] This disclosure relates to the field of image technology, specifically to a text prediction method, apparatus, and electronic device based on a virtual scene. Background Technology

[0002] With the increasing development of artificial intelligence, various functions such as text prediction and human-computer dialogue can be achieved with the help of AI technology. For example, users can conduct human-computer interactive dialogues through text dialogue models: when a user asks a question in voice form, the text dialogue model can automatically output a voice response.

[0003] However, the inventors discovered in the process of developing this invention that the current implementation method can only predict plain text content in the form of dialogues and cannot be applied to more complex text prediction scenarios. Summary of the Invention

[0004] In view of the above problems, this disclosure is made in order to provide a text prediction method, apparatus and electronic device based on virtual scene to overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present disclosure, a text prediction method based on a virtual scene is provided, comprising:

[0006] Obtain the initial text in the virtual scene and determine the text features corresponding to the initial text;

[0007] Obtain the virtual scene image corresponding to the initial text in the virtual scene, and determine the image features of the virtual scene image;

[0008] Based on the image features of the virtual scene image, interactive fusion processing is performed on the text features of the initial text to obtain the text fusion features of the initial text;

[0009] Based on the text fusion features of the initial text, predict subsequent text in the virtual scene.

[0010] In one optional implementation, the initial text in the virtual scene includes: multiple text statements arranged in a preset order; the text features of the initial text include: multiple statement text features arranged in a preset order, wherein each statement text feature corresponds to a text statement;

[0011] Furthermore, the virtual scene image includes: a plurality of statement scene images arranged in a preset order, wherein each text statement corresponds to at least one statement scene image; the image features of the virtual scene image include: a plurality of statement image features arranged in a preset order; wherein each statement image feature corresponds to one statement scene image.

[0012] In one optional implementation, the plurality of text statements arranged in a preset order are: a plurality of game text statements extracted from a plurality of game video frames of the game application according to a preset keyword list;

[0013] The preset order is determined based on the playback order of multiple game video frames corresponding to multiple game text statements; the preset keyword list is used to store game keywords corresponding to game types.

[0014] Furthermore, the scene image of the statement is determined based on the game video frame corresponding to the text statement.

[0015] In one optional implementation, the step of performing interactive fusion processing on the text features of the initial text based on the image features of the virtual scene image to obtain the text fusion features of the initial text includes:

[0016] For any text statement, based on the statement image features of at least one related statement scene image of the text statement, interactive fusion processing is performed on the statement text features of the text statement to obtain the statement fusion features of the text statement.

[0017] The statement fusion features of multiple text statements are arranged in the preset order to obtain the text fusion features of the initial text;

[0018] Wherein, at least one related statement scene image of any text statement includes: a target scene image containing the any text statement, and / or an associated scene image that is related to the any text statement.

[0019] In one optional implementation, the step of performing interactive fusion processing on the statement text features of any given text statement based on statement image features of at least one related statement scene image of the given text statement to obtain the statement fusion features of the given text statement includes:

[0020] For any text statement, a target scene image containing the text statement is determined. Based on the statement image features of the target scene image, intra-sentence fusion processing is performed on the statement text features of the text statement to obtain the intra-sentence fusion features of the text statement.

[0021] Determine the associated text statement corresponding to any of the text statements, and perform inter-sentence fusion processing on the intra-sentence fusion features of the associated text statements based on the intra-sentence fusion features of the associated text statements to obtain the statement fusion features of the any of the text statements.

[0022] In one optional implementation, determining the associated text statement corresponding to any text statement includes:

[0023] Calculate the statement similarity between other text statements and any given text statement, and determine the text statements with a statement similarity greater than a preset similarity threshold as associated text statements corresponding to any given text statement; wherein, the statement similarity is determined based on the text similarity between other text statements and any given text statement, and the image similarity between the statement scene image corresponding to the other text statements and the statement scene image corresponding to the given text statement; or,

[0024] Determine the identification information corresponding to any of the text statements, and determine the associated text statements corresponding to any of the text statements based on the identification information; wherein, the identification information includes: game stage information, and / or chapter / paragraph information.

[0025] In one optional implementation, the text statement contains multiple text characters, and the statement text features of the text statement specifically include: multiple character features, each character feature corresponding to a text character;

[0026] The intra-sentence fusion features of any text statement specifically include: multiple intra-sentence character features, and each intra-sentence character feature corresponds to a text character;

[0027] The intra-sentence character features of each text character are calculated in the following way:

[0028] Calculate the character relevance between the character features of the text characters and the sentence image features of the target scene image, and set the character fusion weights based on the character relevance;

[0029] Based on the character fusion weight, the sentence image features of the target scene image are weighted and fused with the character features to obtain the intra-sentence character features of the text characters.

[0030] In one alternative implementation, the character characteristics of any text character in any text statement are obtained in the following way:

[0031] Obtain the initial character vector for each text character in the text statement;

[0032] For any text character in the text statement, the initial character vector of the any text character is subjected to intra-sentence interactive fusion processing with the initial character vectors of the other text characters in the text statement to obtain the fused character vector of the any text character; wherein, the fused character vector of the any text character is used to characterize the character features of the any text character.

[0033] In one optional implementation, the associated text statement corresponding to any one of the text statements includes: a first type of associated text statement whose statement similarity to any one of the text statements is greater than a preset first similarity threshold, and a second type of associated text statement whose statement similarity to any one of the text statements is not greater than the preset first similarity threshold.

[0034] The step of performing inter-sentence fusion processing on the intra-sentence fusion features of any text statement based on the intra-sentence fusion features of the associated text statements to obtain the statement fusion features of any text statement specifically includes:

[0035] For any text character in any text statement, perform a first inter-sentence fusion process based on the first type of associated text statements to obtain the first inter-sentence character feature of the text character; perform a second inter-sentence fusion process based on the second type of associated text statements to obtain the second inter-sentence character feature of the text character.

[0036] Based on the character features between the first and second sentences, determine the character fusion features of any text character;

[0037] The statement fusion feature of any text statement is determined based on the character fusion features of multiple text characters in any text statement.

[0038] In one optional implementation, the step of performing a first inter-sentence fusion process based on the first type of associated text statements to obtain the first inter-sentence character feature of any text character specifically includes: obtaining the intra-sentence character features of a first number of text characters contained in the first type of associated text statements, and fusing the intra-sentence character features of the first number of text characters with the intra-sentence character features of any text character in any text statement to obtain the first inter-sentence character feature of any text character.

[0039] The step of performing a second inter-sentence fusion process based on the second type of associated text statements to obtain the second inter-sentence character features of any text character specifically includes: obtaining the intra-sentence character features of a second number of text characters contained in the second type of associated text statements, and fusing the intra-sentence character features of the second number of text characters with the intra-sentence character features of any text character in any text statement to obtain the second inter-sentence character features of any text character; wherein, the first number is greater than the second number.

[0040] In one optional implementation, determining the image features of the virtual scene image specifically includes:

[0041] Acquire image data, interactive element data, and / or game stage data of the virtual scene image;

[0042] The image data, interactive element data, and / or game stage data are vectorized to obtain the image features of the virtual scene image.

[0043] In one optional implementation, predicting subsequent text in the virtual scene based on the text fusion features of the initial text includes:

[0044] Obtain the image features of subsequent scene images, and input the image features of the subsequent scene images into the text prediction model;

[0045] The text prediction model obtains the subsequent text predicted by the subsequent scene image based on the image features and the text fusion features of the initial text.

[0046] In one optional implementation, the virtual scene is a game scene in a game application, and the image features for obtaining subsequent scene images include:

[0047] Display multiple candidate scene images;

[0048] In response to an image selection command triggered by a user for any candidate scene image, the image features of subsequent scene images are obtained based on the image identifier contained in the image selection command.

[0049] In one alternative implementation, the text prediction model is trained in the following manner:

[0050] Obtain initial samples in the virtual scene and determine the sample features corresponding to the initial samples;

[0051] Obtain a sample scene image in the virtual scene corresponding to the initial sample, and determine the image features of the sample scene image;

[0052] Based on the image features of the sample scene image, an interactive fusion process is performed on the sample features of the initial sample to obtain the sample fusion features of the initial sample;

[0053] The text prediction model is trained based on the sample fusion features of the initial samples and the pre-annotated label text of the initial samples.

[0054] In one optional implementation, the step of performing interactive fusion processing on the text features of the initial text to obtain the text fusion features of the initial text specifically includes:

[0055] The text features of the initial text are adjusted using the image features of the virtual scene image, and the adjusted text features are used as the text fusion features of the initial text.

[0056] Among them, the image features of the virtual scene image are image modal features, the text features of the initial text are text modal features, and the text fusion features of the initial text are multimodal features that fuse the image modal features and the text modal features.

[0057] According to another aspect of the present disclosure, a text prediction device based on a virtual scene is provided, comprising:

[0058] The text acquisition module is adapted to acquire initial text in the virtual scene and determine text features corresponding to the initial text;

[0059] An image acquisition module is adapted to acquire a virtual scene image corresponding to the initial text in the virtual scene and determine the image features of the virtual scene image;

[0060] The fusion module is adapted to perform interactive fusion processing on the text features of the initial text based on the image features of the virtual scene image, so as to obtain the text fusion features of the initial text;

[0061] The prediction module is adapted to predict subsequent text in the virtual scene based on the text fusion features of the initial text.

[0062] In one optional implementation, the initial text in the virtual scene includes: multiple text statements arranged in a preset order; the text features of the initial text include: multiple statement text features arranged in a preset order, wherein each statement text feature corresponds to a text statement;

[0063] Furthermore, the virtual scene image includes: a plurality of statement scene images arranged in a preset order, wherein each text statement corresponds to at least one statement scene image; the image features of the virtual scene image include: a plurality of statement image features arranged in a preset order; wherein each statement image feature corresponds to one statement scene image.

[0064] In one optional implementation, the plurality of text statements arranged in a preset order are: a plurality of game text statements extracted from a plurality of game video frames of the game application according to a preset keyword list;

[0065] The preset order is determined based on the playback order of multiple game video frames corresponding to multiple game text statements; the preset keyword list is used to store game keywords corresponding to game types.

[0066] Furthermore, the scene image of the statement is determined based on the game video frame corresponding to the text statement.

[0067] In one alternative implementation, the fusion module is specifically adapted to:

[0068] For any text statement, based on the statement image features of at least one related statement scene image of the text statement, interactive fusion processing is performed on the statement text features of the text statement to obtain the statement fusion features of the text statement.

[0069] The statement fusion features of multiple text statements are arranged in the preset order to obtain the text fusion features of the initial text;

[0070] Wherein, at least one related statement scene image of any text statement includes: a target scene image containing the any text statement, and / or an associated scene image that is related to the any text statement.

[0071] In one alternative implementation, the fusion module is specifically adapted to:

[0072] For any text statement, a target scene image containing the text statement is determined. Based on the statement image features of the target scene image, intra-sentence fusion processing is performed on the statement text features of the text statement to obtain the intra-sentence fusion features of the text statement.

[0073] Determine the associated text statement corresponding to any of the text statements, and perform inter-sentence fusion processing on the intra-sentence fusion features of the associated text statements based on the intra-sentence fusion features of the associated text statements to obtain the statement fusion features of the any of the text statements.

[0074] In one alternative implementation, the fusion module is specifically adapted to:

[0075] Calculate the statement similarity between other text statements and any given text statement, and determine the text statements with a statement similarity greater than a preset similarity threshold as associated text statements corresponding to any given text statement; wherein, the statement similarity is determined based on the text similarity between other text statements and any given text statement, and the image similarity between the statement scene image corresponding to the other text statements and the statement scene image corresponding to the given text statement; or,

[0076] Determine the identification information corresponding to any of the text statements, and determine the associated text statements corresponding to any of the text statements based on the identification information; wherein, the identification information includes: game stage information, and / or chapter / paragraph information.

[0077] In one optional implementation, the text statement contains multiple text characters, and the statement text features of the text statement specifically include: multiple character features, each character feature corresponding to a text character;

[0078] The intra-sentence fusion features of any text statement specifically include: multiple intra-sentence character features, and each intra-sentence character feature corresponds to a text character;

[0079] The intra-sentence character features of each text character are calculated in the following way:

[0080] Calculate the character relevance between the character features of the text characters and the sentence image features of the target scene image, and set the character fusion weights based on the character relevance;

[0081] Based on the character fusion weight, the sentence image features of the target scene image are weighted and fused with the character features to obtain the intra-sentence character features of the text characters.

[0082] In one alternative implementation, the character characteristics of any text character in any text statement are obtained in the following way:

[0083] Obtain the initial character vector for each text character in the text statement;

[0084] For any text character in the text statement, the initial character vector of the any text character is subjected to intra-sentence interactive fusion processing with the initial character vectors of the other text characters in the text statement to obtain the fused character vector of the any text character; wherein, the fused character vector of the any text character is used to characterize the character features of the any text character.

[0085] In one optional implementation, the associated text statement corresponding to any one of the text statements includes: a first type of associated text statement whose statement similarity to any one of the text statements is greater than a preset first similarity threshold, and a second type of associated text statement whose statement similarity to any one of the text statements is not greater than the preset first similarity threshold.

[0086] The fusion module is specifically adapted to:

[0087] For any text character in any text statement, perform a first inter-sentence fusion process based on the first type of associated text statements to obtain the first inter-sentence character feature of the text character; perform a second inter-sentence fusion process based on the second type of associated text statements to obtain the second inter-sentence character feature of the text character.

[0088] Based on the character features between the first and second sentences, determine the character fusion features of any text character;

[0089] The statement fusion feature of any text statement is determined based on the character fusion features of multiple text characters in any text statement.

[0090] In one optional implementation, the fusion module is specifically adapted to: obtain the intra-sentence character features of a first number of text characters contained in the first type of associated text statement, and fuse the intra-sentence character features of the first number of text characters with the intra-sentence character features of any text character in any text statement to obtain the first inter-sentence character features of any text character;

[0091] Obtain the intra-sentence character features of a second number of text characters contained in the second type of associated text statement, and fuse the intra-sentence character features of the second number of text characters with the intra-sentence character features of any text character in any text statement to obtain the second inter-sentence character features of any text character; wherein, the first number is greater than the second number.

[0092] In one alternative implementation, the image acquisition module is specifically adapted to:

[0093] Acquire image data, interactive element data, and / or game stage data of the virtual scene image;

[0094] The image data, interactive element data, and / or game stage data are vectorized to obtain the image features of the virtual scene image.

[0095] In one alternative implementation, the prediction module is specifically adapted to:

[0096] Obtain the image features of subsequent scene images, and input the image features of the subsequent scene images into the text prediction model;

[0097] The text prediction model obtains the subsequent text predicted by the subsequent scene image based on the image features and the text fusion features of the initial text.

[0098] In one optional implementation, the virtual scene is a game scene in a game application, and the prediction module is specifically adapted to:

[0099] Display multiple candidate scene images;

[0100] In response to an image selection command triggered by a user for any candidate scene image, the image features of subsequent scene images are obtained based on the image identifier contained in the image selection command.

[0101] In one alternative implementation, the fusion module is specifically adapted to:

[0102] The text features of the initial text are adjusted using the image features of the virtual scene image, and the adjusted text features are used as the text fusion features of the initial text.

[0103] Among them, the image features of the virtual scene image are image modal features, the text features of the initial text are text modal features, and the text fusion features of the initial text are multimodal features that fuse the image modal features and the text modal features.

[0104] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0105] The memory is used to store at least one executable instruction that causes the processor to perform the text prediction method as described above.

[0106] According to another aspect of this disclosure, a computer storage medium is provided, the storage medium storing at least one executable instruction that causes a processor to perform the text prediction method as described above.

[0107] In the embodiments of this disclosure, given initial text in a virtual scene, subsequent text in the virtual scene can be automatically predicted. During the prediction process, interactive fusion processing is performed on the text features of the virtual scene image corresponding to the initial text, thereby obtaining text fusion features of the initial text. These text fusion features are then used to predict subsequent text. Compared to directly using the text features of the initial text for prediction, the text fusion features in this embodiment fuse both text and image features. Therefore, the text fusion features are multimodal features (containing at least text and image modalities), which can more comprehensively reflect the characteristics of the virtual scene, resulting in more accurate and efficient prediction of subsequent text.

[0108] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0109] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0110] Figure 1 A flowchart illustrating a text prediction method based on a virtual scene provided in one embodiment of this disclosure is shown;

[0111] Figure 2 The diagram illustrates a training method for a text prediction model.

[0112] Figure 3 A flowchart illustrating a text prediction method based on a virtual scene, provided in a specific example of this disclosure, is shown.

[0113] Figure 4 A schematic diagram of the inter-sentence fusion process is shown;

[0114] Figure 5 A flowchart of a text prediction device based on a virtual scene provided in one embodiment of this disclosure is shown;

[0115] Figure 6 A schematic diagram of the structure of an electronic device provided in yet another embodiment of this disclosure is shown. Detailed Implementation

[0116] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0117] Figure 1 A flowchart illustrating a text prediction method based on a virtual scene according to an embodiment of this disclosure is shown. Figure 1 As shown, the method includes:

[0118] Step S110: Obtain the initial text in the virtual scene and determine the text features corresponding to the initial text.

[0119] In this embodiment, the virtual scene includes various scenes presented on electronic screens, such as game scenes, virtual reality scenes, and human-computer interaction scenes. The initial text in the virtual scene typically includes known text already presented to the user within the virtual scene. Therefore, the initial text serves as a known condition for predicting subsequent text. In practice, the initial text in the virtual scene can be text already presented in the scene interface of the virtual scene. Specifically, the initial text in the virtual scene can be either all the text already displayed in the virtual scene or only a portion of the text already displayed. Where the initial text only contains the portion of text already displayed in the virtual scene, the initial text can be obtained by filtering from all the text already displayed in the virtual scene using preset filtering conditions.

[0120] Here, the text features corresponding to the initial text refer to the content that can characterize the features of the initial text, obtained by performing feature extraction operations on the initial text. For example, in one optional implementation, the initial text can be vectorized, converting it into text vectors, and then using these text vectors as text features. In short, this application does not limit the specific method of obtaining text features, as long as it can effectively characterize the features of the text.

[0121] Step S120: Obtain the virtual scene image corresponding to the initial text in the virtual scene, and determine the image features of the virtual scene image.

[0122] In this context, the virtual scene image corresponding to the initial text refers to a virtual scene image that is related to the initial text. For example, a virtual scene image corresponding to the initial text includes either a virtual scene image containing the initial text or a virtual scene image containing the context of the initial text. For instance, in virtual scenes such as games, not only are image contents included, but also text contents displayed on top of the image contents. Accordingly, if the text content is determined to be the initial text, the image containing that text content is a virtual scene image. The image features of the virtual scene image can be obtained through feature extraction methods, such as extraction using a pre-trained neural network. This application does not limit the specific connotation and extraction method of the image features, as long as they can characterize the features of the image.

[0123] Step S130: Based on the image features of the virtual scene image, perform interactive fusion processing on the text features of the initial text to obtain the text fusion features of the initial text.

[0124] In one optional implementation, this step is specifically implemented in the following way: the text features of the initial text are adjusted using the image features of the virtual scene image, and the adjusted text features are used as the text fusion features of the initial text; wherein, the image features of the virtual scene image are image modal features, and the text features of the initial text are text modal features; the text fusion features of the initial text are multimodal features that fuse the image modal features and the text modal features.

[0125] The initial text features are unimodal features, used to characterize the text's morphological features. The virtual scene image features are also unimodal features, used to characterize the image's morphological features. The interactive fusion processing aims to integrate the textual features of the text morphology with the image features of the image morphology, using both to characterize the initial text's features. Specifically, interactive fusion processing involves adjusting the initial text features using the image features of the virtual scene image, so that the adjusted fused text features not only reflect the initial text's features but also the image features of the virtual scene image, thus providing a more comprehensive description of the initial text's characteristics using the virtual scene image.

[0126] In practice, the initial text contained within a virtual scene image is usually closely related to the image content of the virtual scene. For example, if the virtual scene image is an indoor image of a classroom, the text content is usually related to classroom content; if the virtual scene image is an outdoor image of a playground, the text content is usually related to outdoor activities. Therefore, by using multimodal text fusion features, the semantic features of the initial text can be represented more comprehensively and accurately.

[0127] Step S140: Based on the text fusion features of the initial text, predict the subsequent text in the virtual scene.

[0128] Specifically, a pre-trained text prediction model can be used to predict subsequent text corresponding to the text fusion features of the initial text. This application does not limit the specific prediction method. Subsequent text in a virtual scene refers to text content that has a semantic connection with the initial text in the virtual scene.

[0129] In this embodiment, the initial text in the virtual scene can be one or more statements, one or more paragraphs, etc. Correspondingly, subsequent text can be at least one statement or at least one paragraph that follows the initial text.

[0130] Therefore, in the embodiments of this disclosure, given the initial text in a virtual scene, subsequent text in the virtual scene can be automatically predicted. During the prediction process, interactive fusion processing is performed on the text features of the virtual scene image corresponding to the initial text, thereby obtaining the text fusion features of the initial text. This text fusion feature is then used to predict subsequent text. Compared to directly using the text features of the initial text for prediction, the text fusion features in this embodiment fuse both text and image features. Therefore, the text fusion features are multimodal features (containing at least text and image modalities), which can more comprehensively reflect the characteristics of the virtual scene, thus making the prediction results of subsequent text more accurate and efficient.

[0131] In the above embodiments, given the initial text in the virtual scene, subsequent text in the virtual scene can be automatically predicted. During the prediction process, interactive fusion processing is performed on the text features of the virtual scene image corresponding to the initial text, thereby obtaining the text fusion features of the initial text. This text fusion feature is then used to predict subsequent text. Compared to directly using the text features of the initial text for prediction, the text fusion features in this embodiment fuse both text and image features. Therefore, the text fusion features are multimodal features (containing at least text and image modalities), which can more comprehensively reflect the characteristics of the virtual scene, thus making the prediction results of subsequent text more accurate and efficient.

[0132] In one optional implementation, to improve the prediction accuracy of subsequent text, prediction is performed based on multiple text statements and multiple scene images. Accordingly, the initial text in the virtual scene mentioned above includes multiple text statements arranged in a preset order. For example, the initial text can be multiple text statements arranged according to their order of appearance in the virtual scene. The text features of the initial text include multiple statement text features arranged in a preset order, where each statement text feature corresponds to one text statement. For example, each text statement can be converted into its corresponding statement text feature sequentially, and the text features of multiple text statements can be arranged sequentially to obtain the text features of the initial text. The statement text features of each text statement can be generated through various methods such as word vectors, and this application does not limit this.

[0133] In the above method, to adapt to multiple text statements, the virtual scene image includes: multiple statement scene images arranged in a preset order. Each text statement corresponds to at least one statement scene image. For example, a text statement can correspond to one or more statement scene images; for instance, if a long text statement appears in two virtual scene images, it corresponds to two different statement scene images. Alternatively, different text statements can correspond to the same statement scene image; for example, multiple statements may be contained in a single statement scene image. This application does not limit the specific correspondence between text statements and statement scene images. Since multiple text statements are arranged in an ordered manner, the multiple statement scene images are also arranged in an ordered manner. The image features of the virtual scene image include: multiple statement image features arranged in a preset order; each statement image feature corresponds to one statement scene image. Specifically, the statement image features of each statement scene image can be obtained through various methods such as feature extraction networks. In summary, when the initial text includes multiple ordered text statements and the virtual scene image also contains multiple ordered statement scene images, the prediction can be further improved by combining the statement order of the multiple text statements and the image order of the multiple statement scene images.

[0134] In one optional implementation, the text statements in the initial text, arranged in a preset order, can be filtered according to the specific application scenario to adapt to the needs of different scenarios. For example, in a game application scenario, to meet the actual needs of different types of games, a keyword list is pre-set, storing game keywords corresponding to the game type. For instance, because the game world differs from the real world, there are some special words in the game world that are different from those in the real world, which makes it impossible for traditional text prediction models to learn the relevant information of such special words. For example, in the game world, there are some game rules that are different from those in the real world, thus giving the text statements in the game world significant characteristics of the game world. Therefore, to improve the accuracy of this embodiment in game applications, a keyword list is pre-configured according to the relevant settings of the game world, storing feature words that are highly related to the game world. Correspondingly, the multiple text statements arranged in a preset order mentioned above are: multiple game text statements extracted from multiple game video frames of the game application according to the preset keyword list. For example, extracting text statements or text paragraphs that match the keyword list from the large amount of text content contained in multiple game video frames to generate the initial text can improve the prediction accuracy of subsequent text. The aforementioned preset order can be determined based on the playback order of multiple game video frames corresponding to multiple game text statements: the order of game text statements in earlier-played game video frames precedes the order of game text statements in later-played game video frames. The aforementioned statement scene images are determined based on the game video frames corresponding to the text statements. For example, suppose there are three text statements: the first text statement appearing in the first game video frame, the second text statement appearing in the second game video frame, and the third text statement appearing in the third game video frame. Accordingly, the statement scene image corresponding to the first text statement is the first game video frame, the statement scene image corresponding to the second text statement is the second game video frame, and the statement scene image corresponding to the third text statement is the third game video frame.

[0135] In one optional implementation, when the initial text contains multiple text statements, the text fusion feature of the initial text is obtained by sequentially arranging the statement fusion features of the multiple text statements, and the statement fusion feature of each text statement is generated based on at least one related statement scene image of that text statement. Specifically, when performing interactive fusion processing on the text features of the initial text based on the image features of the virtual scene image to obtain the text fusion feature of the initial text, it is implemented in the following way: First, for any text statement, based on the statement image features of at least one related statement scene image of any text statement, interactive fusion processing is performed on the statement text features of any text statement to obtain the statement fusion feature of any text statement. Then, the statement fusion features of the multiple text statements are arranged in a preset order to obtain the text fusion feature of the initial text. The purpose of performing interactive fusion processing on the statement text features of any text statement is to enable the statement fusion feature of any text statement to simultaneously reflect the features of both the text modality and the image modality, that is, for any text statement, the image features of the statement scene image related to that text statement are incorporated into its statement text features. Correspondingly, at least one related statement scene image for any text statement refers to a statement scene image that interacts (i.e., is mutually related) with any text statement. For example, since a virtual scene contains multiple text statements and multiple statement scene images, a related statement scene image for a text statement typically refers to the statement scene image containing the text statement and the statement scene image containing the context of the text statement. For example, in one implementation, at least one related statement scene image for any text statement includes: a target scene image containing the text statement, and / or an associated scene image that is related to the text statement. The target scene image containing the text statement is the scene image containing the text statement, and the associated scene image is the scene image containing the text statement that is semantically related to the text statement. In practical processing, the text features of a text statement can be fused based solely on the image features of the target scene image containing any text statement, enabling image-text fusion within the same text statement. Alternatively, the text features of a text statement can be fused based on the image features of the target scene image containing any text statement and the image features of related scene images that are associated with any text statement, enabling image-text fusion of a text statement with the image features of multiple scene images, further improving the comprehensiveness of the fusion.

[0136] In one optional implementation, for any text statement, intra-sentence fusion processing is performed first, followed by inter-sentence fusion processing. Intra-sentence fusion processing ensures effective fusion between features within the same sentence, while inter-sentence fusion processing ensures effective fusion between multiple sentences, thereby improving the accuracy of the fused features. Specifically, when performing interactive fusion processing on the text features of any text statement based on the text image features of at least one related scene image of that text statement to obtain the fused features of that text statement, this is achieved in the following way:

[0137] First, for any given text statement, a target scene image containing that text statement is identified. Based on the statement image features of the target scene image, intra-sentence fusion processing is performed on the text features of any given text statement to obtain the intra-sentence fused features of that text statement. The target scene image containing any given text statement can be one or more. For example, when a text statement is long and its beginning and end are located in different target scene images, intra-sentence fusion processing can be performed by averaging the image features of multiple target scene images using pre-defined operations. Through intra-sentence fusion processing, the text features of each text statement are fully integrated with the statement image features of the corresponding target scene image, thus ensuring that the intra-sentence fused features of each text statement contain both textual and image information.

[0138] Then, the associated text statements corresponding to any given text statement are identified. Based on the intra-sentence fusion features of the associated text statements, inter-sentence fusion processing is performed on the intra-sentence fusion features of any given text statement to obtain the statement fusion features of any given text statement. Associated text statements refer to text statements that are related to any given text statement; specifically, they can be context statements corresponding to any given text statement, or statements that are semantically related to any given text statement. In practice, all text statements other than any given text statement can be identified as associated text statements to ensure the comprehensiveness of feature fusion; alternatively, specific text statements among the remaining text statements can be identified as associated text statements to ensure the specificity of feature fusion. Since the intra-sentence fusion features of each text statement integrate multimodal features from both text and image perspectives, by performing inter-sentence fusion processing on the intra-sentence fusion features of any given text statement and the intra-sentence fusion features of other associated text statements, multimodal features integrating text-image features from multiple sentences can be obtained. Therefore, after the above processing, the statement fusion features of each text statement integrate text-image features within the statement as well as text-image features between multiple statements.

[0139] The associated text statement corresponding to any text statement can be determined in several ways:

[0140] In one optional implementation, associated text statements are determined based on the similarity between text statements. Specifically, the statement similarity between other text statements and any given text statement is calculated, and text statements with a similarity greater than a preset similarity threshold are identified as associated text statements corresponding to any given text statement. The statement similarity can be determined based on the text similarity between other text statements and any given text statement, as well as the image similarity between the scene image corresponding to the other text statements and the scene image corresponding to the given text statement. Therefore, by using both text similarity and image similarity to characterize statement similarity, the calculation results of statement similarity can be made more accurate. This method can identify semantically related statements as associated text statements, thereby improving the targeting of subsequent fusion.

[0141] In another optional implementation, associated text statements are determined based on their identifier information. Specifically, the identifier information corresponding to any given text statement is determined, and text statements with the same identifier information are identified as associated text statements corresponding to that text statement. The identifier information may include game stage information and / or chapter / paragraph information. For example, game stage information describes the current stage or progress of the game; statements and images at the same stage or progress are usually highly correlated. Therefore, text statements at the same game stage can be considered as associated text statements. This method eliminates the need for similarity calculations and can quickly determine associated text statements directly based on the identifier information, thus improving processing speed.

[0142] In practice, the two methods for determining related text statements mentioned above can be used individually or in combination, and this application does not limit this.

[0143] In another optional implementation, the text statement contains multiple text characters, which can be single Chinese characters, English words, symbols, etc. Accordingly, to comprehensively describe the characteristics of each text character, the statement text features specifically include: multiple character features, each corresponding to a text character. Furthermore, the intra-sentence fusion features of any text statement specifically include: multiple intra-sentence character features, each corresponding to a text character. Therefore, for each text character in the text statement, the character features of that text character are first calculated, and then the statement image features of the target scene image are integrated into the character features of each text character, thus obtaining the intra-sentence character features of each text character. Therefore, the intra-sentence character features are image-text hybrid features that incorporate the image.

[0144] In one optional implementation, the intra-sentence character features of each text character are calculated as follows: First, the character relevance between the text character features and the sentence image features of the target scene image is calculated, and character fusion weights are set based on the character relevance. Character relevance can be calculated using various methods such as similarity algorithms and inner product algorithms, as long as it can represent the degree of correlation between the character features and the sentence image features. For example, the greater the character relevance between the text character features and the sentence image features of the target scene image, the greater the character fusion weight; conversely, the smaller the character relevance, the smaller the character fusion weight. Then, based on the character fusion weights, the sentence image features of the target scene image and the character features are weighted and fused to obtain the intra-sentence character features of the text character. This method allows for the calculation of the relevance between each character and the target scene image, and enables image-text fusion processing character by character to ensure that characters with higher relevance incorporate more image features, thus making the final intra-sentence character features of the text character more accurate.

[0145] In another alternative implementation, considering the potential inaccuracies in the character features of a single text character—for example, in the presence of polyphonic characters—the semantic content of a character cannot be accurately determined based on a single text character, resulting in inaccurate generated character features. To address this issue, intra-sentence interactive fusion processing can be performed on each text character to incorporate the features of related characters in the context into the character features of each text character. Specifically, the character features of any text character in any text sentence are obtained as follows: First, the initial character vector of each text character in the text sentence is obtained, which can be achieved through various vectorization methods such as word vectors. Then, for any text character in the text sentence, the initial character vector of that text character is inter-sentence interactively fused with the initial character vectors of the other text characters in the text sentence to obtain the fused character vector of that text character. The fused character vector of any text character is used to represent the character features of that text character after incorporating context semantic information. The intra-sentence interactive fusion processing can be implemented through multi-head attention mechanisms, etc., and this application does not limit the specific details.

[0146] In one alternative implementation, the inventors discovered that if the range of related text statements is too large, the computational load for inter-sentence fusion processing will be excessive, and irrelevant noise will be easily introduced; if the range of related text statements is too small, the features between multiple sentences will not be fully fused, reducing accuracy. To solve the above problems, the related text statements corresponding to any text statement are further divided into two types, and different inter-sentence fusion processing methods are set for each type of related text statement. For example, the related text statements corresponding to any text statement include: a first type of related text statements whose statement similarity to any text statement is greater than a preset first similarity threshold, and a second type of related text statements whose statement similarity to any text statement is not greater than the preset first similarity threshold. The specific value of the preset first similarity threshold can be flexibly set according to the actual scenario; the first type of related text statements are those with a high degree of relevance to any text statement, and the second type of related text statements are those with a low degree of relevance to any text statement. Accordingly, when performing inter-sentence fusion processing on the intra-sentence fusion features of any text statement based on the intra-sentence fusion features of related text statements to obtain the sentence fusion features of any text statement, this is specifically achieved in the following way:

[0147] First, for the intra-sentence character features of any text character in any text statement, on the one hand, a first inter-sentence fusion process is performed based on the first type of associated text statements to obtain the first inter-sentence character features of any text character; on the other hand, a second inter-sentence fusion process is performed based on the second type of associated text statements to obtain the second inter-sentence character features of any text character. The fusion depth of the first inter-sentence fusion process is greater than the fusion depth of the second inter-sentence fusion process. In this way, the two types of associated text statements can be integrated into the intra-sentence character features of any text character in any text statement through different inter-sentence fusion processing methods.

[0148] Then, based on the character features between the first and second sentences, the character fusion features of any text character are determined. Specifically, the character features between the first and second sentences can be weighted to obtain the character fusion features of the text character. Thus, the character fusion features of each text character simultaneously incorporate features from two types of related text statements, and the integration method for each type of related text statement is different. This method ensures that the character fusion features of any text character in any text statement fully incorporate the graphic and textual features of related statements. Furthermore, using different feature integration methods for statements with different degrees of relevance avoids noise interference and improves the comprehensiveness of the features.

[0149] Finally, based on the character fusion features of multiple text characters in any given text statement, the statement fusion features of that text statement are determined. Specifically, the character fusion features of each text character are arranged sequentially according to their order of appearance to obtain the statement fusion features of the text statement.

[0150] The first and second sentence fusion processes mentioned above aim to employ fusion algorithms with different fusion depths for sentences with varying degrees of relevance, so that sentences with high relevance are fused more deeply, while sentences with low relevance are fused less deeply. Any fusion processing method that can achieve this goal can be applied to this application, and no specific details are limited.

[0151] For example, in one specific implementation, the number of text characters to be merged in the first inter-sentence fusion process and the second inter-sentence fusion process are different. Specifically, when performing the first inter-sentence fusion process based on the first type of associated text statements to obtain the first inter-sentence character features of any text character, it is implemented as follows: Obtain the intra-sentence character features of a first number of text characters contained in the first type of associated text statements; merge the intra-sentence character features of the first number of text characters with the intra-sentence character features of any text character in any text statement to obtain the first inter-sentence character features of any text character. When performing the second inter-sentence fusion process based on the second type of associated text statements to obtain the second inter-sentence character features of any text character, it is implemented as follows: Obtain the intra-sentence character features of a second number of text characters contained in the second type of associated text statements; merge the intra-sentence character features of the second number of text characters with the intra-sentence character features of any text character in any text statement to obtain the second inter-sentence character features of any text character; wherein, the first number is greater than the second number.

[0152] For example, in a specific example, the first inter-sentence character feature of any text character is obtained as follows: The intra-sentence character features of all text characters contained in the first type of associated text statement are obtained; the intra-sentence character features of each text character in the first type of associated text statement are then fused with the intra-sentence character features of any text character in any text statement to obtain the first inter-sentence character feature of any text character. The second inter-sentence character feature of any text character is obtained as follows: The intra-sentence character features of a preset text character contained in the second type of associated text statement are obtained; the intra-sentence character features of the preset text character are then fused with the intra-sentence character features of any text character in any text statement to obtain the second inter-sentence character feature of any text character. Here, the preset text character contained in the second type of associated text statement can be the first character in the second type of associated text statement, or it can be a special marker character in the vectorized representation corresponding to the second type of associated text statement, such as a separator character (SEP). In short, the preset text character is preferably a character that can characterize the statement features of the second type of associated text statement. By using the above method, the intra-sentence character features of all text characters contained in the first type of associated text statement can be fused, which can fully guarantee the comprehensiveness of feature fusion. For the intra-sentence character features of the preset text characters contained in the second type of associated text statement, the noise introduced by fusing characters with little correlation can be reduced, and the targeting of the fusion process can be improved.

[0153] In one optional implementation, the image features of the virtual scene image can be extracted using various methods such as feature extraction matrices and neural networks. Furthermore, in addition to the image data content, the image features can further include auxiliary information such as interactive element data (e.g., interactive entry points in the game) and game stage data. Accordingly, when determining the image features of the virtual scene image, the image data, interactive element data, and / or game stage data of the virtual scene image are acquired; these data are then vectorized to obtain the image features of the virtual scene image. For example, the interactive element data and / or game stage data involved in the image can be described through a separate image channel.

[0154] In one optional implementation, when predicting subsequent text in a virtual scene based on the text fusion features of the initial text, the prediction can be further combined with the image features of the subsequent scene image: First, the image features of the subsequent scene image are obtained and input into the text prediction model. Then, the subsequent text predicted by the text prediction model based on the image features of the subsequent scene image and the text fusion features of the initial text is obtained. The subsequent scene image can be an image that appears after the virtual scene image corresponding to the initial text; correspondingly, the predicted subsequent text is used to display in the subsequent scene image. In a specific implementation, the image features of the subsequent scene image can be concatenated after the text fusion features of the initial text to predict the subsequent text.

[0155] In one alternative implementation, the virtual scene is a game scene within a game application. The image features of subsequent scene images are obtained as follows: First, multiple candidate scene images are displayed; then, in response to a user's image selection command triggered for any candidate scene image, the image features of the subsequent scene image are obtained based on the image identifier contained in the image selection command. In this implementation, the virtual scene image corresponding to the initial text is played first, i.e., the beginning of the game is played first. Then, multiple selectable candidate scene images are displayed to the user. Each candidate scene image is an image that follows the virtual scene image corresponding to the initial text, and each candidate scene image has a selection entry point. The user can trigger an image selection command through the selection entry point to select an image from the multiple candidate scene images as the subsequent scene image, and then generate the corresponding subsequent text based on the subsequent scene image. This method is particularly suitable for scenarios where, given the game's beginning and selectable subsequent scene images, it is necessary to predict subsequent text.

[0156] The text prediction model mentioned above can be trained in several ways. For example, Figure 2 The diagram illustrates a training method for a text prediction model, such as... Figure 2 As shown, the training process specifically includes the following steps:

[0157] Step S210: Obtain the initial sample in the virtual scene and determine the sample features corresponding to the initial sample.

[0158] The initial samples are the text content contained in the sample scene images of the virtual scene. The sample features of the initial samples are the text features of the text content.

[0159] Step S220: Obtain the sample scene image corresponding to the initial sample in the virtual scene, and determine the image features of the sample scene image.

[0160] The method for determining the image features of the sample scene images is similar to that for determining the image features of the virtual scene images, and will not be repeated here.

[0161] Step S230: Based on the image features of the sample scene image, perform interactive fusion processing on the sample features of the initial sample to obtain the sample fusion features of the initial sample.

[0162] The specific method for determining the sample fusion features of the initial sample is similar to that for determining the text fusion features of the initial text, and will not be repeated here.

[0163] Step S240: Train a text prediction model based on the sample fusion features of the initial samples and the pre-annotated label text of the initial samples.

[0164] Here, the sample features correspond to textual features. Through interactive fusion processing, image-text interactive fusion can be performed to obtain the sample fusion features of the initial sample. Furthermore, the pre-labeled initial sample's tag text can be obtained as follows: extract the first N image frames from the scene images and text content contained in the known game scene; obtain the aforementioned initial sample and sample scene image based on the first N image frames; and obtain the pre-labeled initial sample's tag text based on the text content in the (N+1)th image frame, where N is a natural number. Therefore, training based on a known game scene can yield the aforementioned text prediction model. The specific implementation details of performing interactive fusion processing on the sample features can be found in the above description of performing interactive fusion processing on the text features of the initial text. In summary, the model training process is similar to the model's inference and prediction process, and will not be elaborated upon here.

[0165] To facilitate understanding, the following specific example will be used to illustrate the detailed implementation of the above embodiments:

[0166] In related technologies, for some narrative-driven games, such as adventure games (AVG), natural language processing (NLP) techniques and generative large-scale models can be used to automatically generate subsequent plots based on the existing game events. For example, in one approach, narrative-driven games mostly rely on manually written storylines; that is, all branches of the game's plot are pre-designed text. Another approach utilizes generative language models in NLP (such as ChatGPT) to automatically predict subsequent plot text based on existing narrative text. While manually written plot text also has branching paths, it is all pre-made, limiting the diversity of story development, and manual writing is time-consuming and labor-intensive. Although generative large-scale language models can engage in reasonable dialogue-level interaction with humans, research shows that ChatGPT cannot integrate well into the game world setting and cannot understand specific scenes within the game world. Therefore, enabling models to understand the large-scale scenes of the game world remains a technical challenge. In addition, there may be many different game scenes, and the scenes may change multiple times during the development of the game's plot. The scenes corresponding to different stages of the game's plot may also be different. Therefore, it is difficult to model fine-grained scene information using only the model, which makes it impossible for the language model to understand the game setting well.

[0167] This example proposes a model for automatically generating narrative text. This model simultaneously models textual and visual information from the game's existing storyline, considering the interaction between textual and visual information as well as the mutual influence between different stages of the game's progression. This enhances the model's understanding of the narrative development within the game scene and generates subsequent narrative text that is more coherent with the existing events. This solution can automatically generate reasonable, coherent, and consistent subsequent narrative content based on the already occurred narrative text in the game.

[0168] Of course, those skilled in the art will understand that this example is based on the generation of plot text in a game. In practice, it can also be used to generate other forms of text in other applications, and this application does not limit the specific application scenario. The text representation mentioned in this example refers to a continuous vector representation representing discrete text information, such as converting a word or sentence into a vector that represents the semantic information of that word or sentence in its vector space. Attention mechanism refers to the process of mutual influence between different representations (e.g., text representations). For example, each word in a sentence has a corresponding word representation, and different word representations may be influenced by the representations of their context words. Encoding refers to converting the original text or image information into a vector representation that a computer can understand, and decoding refers to the model generating human-understandable text information based on the computer-understandable vector representation.

[0169] like Figure 3 As shown, this example specifically includes the following steps:

[0170] Step S301: Obtain multiple text statements arranged in a preset order.

[0171] The multiple text statements arranged in a preset order constitute the initial text in the virtual scene mentioned above. In this example, the multiple text statements arranged in a preset order are stored in a text sequence. This text sequence can be a plot text sequence.

[0172] Suppose that the sequence of plot text in the game is S = (s1, s2, ..., s...). k ), where s i =(w i1 ,w i2 ,...,w i|Si| ) represents the i-th sentence in the plot text that has already occurred, w ij This represents the j-th character in the i-th sentence. In addition, each sentence has corresponding scene image information (i.e., scene image information containing that sentence), denoted as P = (p1, p2, ..., pj). k ), where p i Representative plot texts i The corresponding images. The scene image information for different sentences may be the same or similar. The scene image information mentioned in this example is the sentence scene image corresponding to each text statement mentioned above.

[0173] Step S302: Generate statement text features corresponding to multiple text statements arranged in a preset order. The statement text features of each text statement include an initial character vector of multiple text characters.

[0174] Specifically, the plot text needs to be initialized into a vector representation that the computer can understand. For example, features can be extracted independently from each basic element (such as each text character) in the text statement to obtain an initial character vector for each text character. This initial character vector can be obtained through various methods such as word vectors.

[0175] For example, for plot text information, the plot text sequence S is first processed into the following format:

[0176] [CLS][SEP]w 11 ,w 12 ,...[SEP]w 21 ,w 22 ,...[SEP]...[SEP]w k1 ,w k2,...

[0177] Here, [CLS] and [SEP] are two special characters, representing the beginning of the input and the separator between different sentences, respectively. Then, each character is initialized with its corresponding character representation, resulting in the sentence text features of the following multiple text statements:

[0178] h [CLS] h [SEP] h 11 ,h 12 ,...h [SEP] h 21 ,h 22 ,...h [SEP] ...h [SEP] h k1 ,h k2 ,...

[0179] Among them, h [CLS] and h [SEP] These are the initial character vectors representing the special characters [CLS] and [SEP], respectively. ij Represents the character w ij The initial character vector. Character w ij This refers to the j-th character in the i-th sentence. The feature dimension of the text features for multiple text sentences can be up to 4096 dimensions.

[0180] Step S303: Generate statement image features corresponding to multiple statement scene images arranged in a preset order.

[0181] Specifically, for scene image information (i.e., image information contained in the sentence scene image), image features (i.e., sentence image features) are obtained through feature extraction. For example, for each image p... i The extracted image features are 7×7×2048 dimensions. To facilitate subsequent processing, a fully connected layer can be used to reduce the dimensionality of the image features to 4096 dimensions, making the dimensionality of the image features the same as that of the character features. Image p i The image features obtained after dimensionality reduction are denoted as h. pi .

[0182] Step S304: Perform intra-sentence fusion processing on the initial character vectors of multiple text characters included in the sentence text features of each text sentence to obtain the fused character vector of multiple text characters included in the sentence text features, and use the fused character vector of the text characters as the character features of the text characters.

[0183] Specifically, in order to improve the information interaction between each text character and avoid the problem of inaccurate features caused by a single character not considering the context, in this step, it is necessary to perform intra-sentence interaction fusion processing on the initial character vector of each text character to obtain the fused character vector of each text character, and use the fused character vector as the character feature of the text character.

[0184] For example, for each sentence s in the plot text i =(w i1 ,w i2 ,...), calculate each character w ij The context-dependent representation (i.e., fused character vector) within this sentence allows the representation of each character to be influenced by its context characters, thus providing a more accurate representation of polysemous characters. Specifically, at least one Transformer encoding layer can be used to compute the fused character vector. Each Transformer encoding layer contains a multi-head attention module and a feedforward network module, specifically used to perform the following processing:

[0185]

[0186]

[0187] MultiHeadAttention is used to represent multi-head attention operations, and FFN represents the operations of feedforward network layers.

[0188] The character w represents the result of calculation by the l-layer transformer encoding layer. i2 The sentence's context-dependent representation (i.e., the fused character vector of text characters, also called the character features of text characters). Specifically, an L-layer transformer encoding layer can be used; therefore, for sentence s... i The context-dependent representations of all its characters can be represented as follows:

[0189] Step S305: Calculate the intra-sentence fusion features for each text sentence, wherein the intra-sentence fusion features include multiple intra-sentence character features corresponding to text characters.

[0190] The intra-sentence character features of each text character are obtained by fusing the sentence image features of the target scene image corresponding to the current text sentence.

[0191] For example, although sentence s i Context-dependent representation within the sentence It includes the interaction between different characters within this sentence, but it does not take into account the corresponding plot image p. i Information about the target scene image (i.e., the current text statement's target scene image). Therefore, in this example, scene image information is also incorporated into the narrative text information. Specifically, this can be achieved through the following narrative text-scene image attention (TI-Attention) calculation formula:

[0192]

[0193] in, The image p represents the integration of the image. i Information (i.e.) The representation of the character [SEP] (i.e., the intra-sentence character feature of the character [SEP]). W1, W2, W p All of these are trainable parameters. This represents the degree of association between image information and the contextual representation of the character [SEP] within the sentence. Therefore, the degree of association depends on... and The inner product operation within the string determines the degree of relevance: the larger the inner product value, the greater the relevance. Specifically, W1 and W2 are used to represent the sentence-internal context of the character [SEP]. and image initialization representation They are mapped to the same feature space, and then the correlation between them is calculated using the inner product. This refers to the character correlation between the character features of the text characters mentioned above and the sentence image features of the target scene image. This refers to the character fusion weights set based on character relevance, as mentioned above.

[0194] Similarly, the intra-sentence character features of the remaining characters can be calculated using the following formula:

[0195]

[0196]

[0197] in, The intra-sentence character feature representing the first character in the i-th sentence. The intra-sentence character feature representing the second character in the i-th sentence.

[0198] Step S306: Determine the associated text statement corresponding to any text statement, and perform inter-sentence fusion processing on the intra-sentence fusion features of any text statement based on the intra-sentence fusion features of the associated text statement to obtain the statement fusion features of any text statement.

[0199] In this example, the associated text statements further include a first type of associated text statements and a second type of associated text statements. The first type of associated text statements has a statement similarity greater than a preset first similarity threshold with any other text statement, while the second type of associated text statements has a statement similarity no greater than the preset first similarity threshold with any other text statement. The reason for distinguishing between these two types of associated text statements is to allow for different inter-sentence fusion methods to be applied to statements with different degrees of relevance during subsequent inter-sentence fusion processing. This approach aims to reduce the impact of noise while incorporating as much relevant information as possible.

[0200] Specifically, after the processing in step S305, for each sentence s i It calculates the context-dependent values ​​for each character within the sentence and incorporates s. i The corresponding scene image information contains intra-sentence character features. However, the information between different sentences remains independent. For example, for s i The character w in ij Its intra-sentence character features Although it contains s i The game contains multimodal information (narrative text + scene images), but not multimodal information from other sentences. Therefore, this step implements an interaction mechanism for inter-sentence multimodal information within the game's narrative scene:

[0201] First, the relevance between different sentences is calculated based on multimodal information in different sentences, so as to determine the first-class and second-class related text sentences for any given text statement based on the relevance between sentences. Among them, sentence s i and sentence s k The correlation between them is calculated using the following formula:

[0202]

[0203] in, and Each represents a sentence s i and sentence s k The corresponding [SEP] character is represented. Since the [SEP] character can characterize the features of the entire sentence, the relevance (i.e., sentence similarity) between two sentences can be calculated using the [SEP] character. Of course, other identifying characters in the sentence can also be selected to calculate the relevance; this application does not limit the specific calculation method of the relevance. Representative sentence s i Corresponding target scene image and sentence s kThe correlation (i.e., image similarity) between corresponding target scene images. Based on information from both sentence similarity and image similarity, it is possible to accurately assess whether two sentences are similar. For example, if the character features of the two sentences themselves are similar, and / or the image features of the target scene images corresponding to the two sentences are similar, then the two sentences are considered to be related text sentences of the first type; otherwise, they are considered to be related text sentences of the second type. For example, if s j With s i similarity sim(s i ,s k If the correlation between the two sentences is less than the threshold γ, it indicates that the two sentences are relatively weakly related and belong to the second type of related text statements; if s j With s i similarity sim(s i ,s k If the correlation between the two sentences is greater than or equal to the threshold γ, it indicates that the two sentences are highly correlated and are considered to be related text statements of the first category.

[0204] With target character w ij For example, it is necessary to send the character w ij The feature representation incorporates character features from the first type of related statements and character features from the second type of related statements, respectively. Since the first type of related statements have a stronger correlation, all character features from the first type of related statements are incorporated into the target character 'w'. ij In the feature representation. Because the relevance of the second type of associated statements is relatively weak, only the preset character features in the second type of associated statements are incorporated into the target character w. ij In the feature representation.

[0205] For example, if s j With s i similarity sim(s i ,s k If the correlation between the two sentences is less than the threshold γ, it indicates a weak correlation. In this case, only s is considered. j The [SEP] character in the sentence is incorporated into the character w. ij In the representation.

[0206] If s j With s i similarity sim(s i ,s k If the correlation between the two sentences is greater than or equal to the threshold γ, it indicates a strong correlation between the two sentences. j The representation of all characters in the sentence is incorporated into the character w. ij In the representation.

[0207] Figure 4 A schematic diagram of the inter-sentence fusion process is shown, such as... Figure 4As shown, assume there are n sentences and their corresponding images. Here, p1 is the image information corresponding to the target scene image of the first sentence, p2 is the image information corresponding to the target scene image of the second sentence, and pi is the image information corresponding to the target scene image of the i-th sentence. Accordingly, the intra-sentence character features of the first sentence include: SEP1, w... 11 w 12 w 13 The same logic applies to other sentences.

[0208] Suppose that the first sentence corresponding to p1 is related to sentence s. i The first category of related text sentences has a similarity greater than the threshold γ, while the remaining sentences are related to sentence s. i The similarity scores of all sentences are below the threshold, meaning the remaining sentences are all Class II related text statements. The curved arrow represents the direction towards the character w. ij The first type of related text sentence incorporates intra-sentence character features, with the broken arrow representing the direction towards the character 'w'. ij The second type of related text sentences incorporates intra-sentence character features. It's important to note that the intra-sentence character features of each character in the remaining sentences have already incorporated the corresponding image information, and the intra-sentence character features of characters within each sentence are obtained by fusing the intra-sentence context. That is, the intra-sentence character features of characters within each sentence are already multimodal information. Therefore, after performing inter-sentence fusion processing, the resulting character fusion feature is multimodal information that integrates information from multiple sentences. Specifically, the character w... ij The formula for calculating the character fusion feature is as follows:

[0209]

[0210]

[0211]

[0212] in, For the character w ij Intra-sentence character features, For the character w m The intra-sentence character features.

[0213] W f W o and W p For trainable parameters, a ij Represents the character w ij The integrated information from the remaining highly similar sentences (i.e., the first type of related text sentences) incorporated into r ij Represents the character w ijThe integrated information is derived from the remaining low-similarity sentences (i.e., second-class related text sentences). sgn is the sign function; when sim(s i ,s k When ) is greater than the threshold γ, sgn(max(0, sim(s) i ,s k )-γ)) is 1 otherwise 0. Q, K, and V are trainable parameters. Represents the character w mn and the character w ij The correlation between them. sim(s) i ,s k ) represents sentence s i and sentence s k The correlation between them.

[0214] As can be seen from the formula, for the character w ij Specifically, it is necessary to incorporate the intra-sentence character features of each character in the first type of associated text statements, as well as the intra-sentence character features of the SEP character in the second type of associated text statements. Among these, The sentence-level character features of the character SEP are obtained by sequentially arranging the character fusion features of all characters within a text sentence.

[0215] Step S307: Obtain the image features of the subsequent scene image, and predict the subsequent text based on the image features of the subsequent scene image and the sentence fusion features of the multiple text sentences mentioned above.

[0216] Steps S301 to S306 can be obtained through a text prediction model (e.g., the encoder in a Transformer model). Correspondingly, step S307 can be obtained through a decoder in a Transformer model. Specifically, the character fusion features are calculated for all the characters in the entire plot through the above steps. This character fusion feature contains intra-sentence contextual information, intra-sentence scene image information, and inter-sentence multimodal information. Finally, based on the character fusion features in each sentence and the scene image of the subsequent story (denoted as p), k+1 This generates subsequent text-based plot. In practice, p can be used... k+1 Image features After concatenating the character fusion feature sequences of all characters in the entire statement, the Transformer decoder is then used to generate subsequent text based on the following representation sequence:

[0217]

[0218] The Transformer decoder will generate subsequent story text step by step.

[0219] In summary, this example proposes an automated model for generating plot text. This model simultaneously models the text and image information in the already occurred plot of the game, and considers the interaction between text and image information as well as the mutual influence between different stages of the game process. This improves the model's ability to understand the plot development in the game scene and generates subsequent plot text that is more coherent with the already occurred plot. This application can automatically generate reasonable subsequent plot content that is coherent with the preceding text and conforms to the game background setting based on the plot text that has already occurred in the game. This application proposes a GameGPT model, which mainly consists of five parts, and the five parts have a sequential relationship: (1) the first part: used to realize the representation initialization of plot text information and scene image information; (2) the second part: used to realize the representation interaction between text information within sentences; (3) the third part: used to realize the representation interaction between plot text information and corresponding scene image information; (4) the fourth part: used to realize the representation interaction of multimodal information between sentences; (5) the fifth part: used for decoding and generating subsequent plot.

[0220] This example encodes existing plot text and scene images within a game and automatically predicts subsequent plot text based on a pre-trained text prediction model. The training process for the text prediction model is similar to the inference prediction process. For instance, during training, the initial sample is determined based on the text contained in the first N scene images of the known game plot, and the corresponding label text is determined based on the text contained in the (N+1)th scene image of the game plot. The model is trained using the initial sample, and the convergence condition of the model is judged using a loss function and the label text until a text prediction model meeting the accuracy requirements is obtained. In summary, the text fusion features of the initial text in this example combine multimodal information such as intra-sentence context information, intra-sentence image information, and inter-sentence image-text information. Therefore, the prediction results are more accurate and more consistent with the game world setting. This example further enhances the ability of generative language models to understand the game world setting and can generate subsequent plots that are more consistent with the game scene.

[0221] Figure 5 The diagram illustrates a text prediction device based on a virtual scene according to another embodiment of this disclosure. The text prediction device includes:

[0222] The text acquisition module 51 is adapted to acquire initial text in the virtual scene and determine text features corresponding to the initial text;

[0223] Image acquisition module 52 is adapted to acquire a virtual scene image corresponding to the initial text in the virtual scene and determine the image features of the virtual scene image;

[0224] The fusion module 53 is adapted to perform interactive fusion processing on the text features of the initial text based on the image features of the virtual scene image, so as to obtain the text fusion features of the initial text;

[0225] The prediction module 54 is adapted to predict subsequent text in the virtual scene based on the text fusion features of the initial text.

[0226] In one optional implementation, the initial text in the virtual scene includes: multiple text statements arranged in a preset order; the text features of the initial text include: multiple statement text features arranged in a preset order, wherein each statement text feature corresponds to a text statement;

[0227] Furthermore, the virtual scene image includes: a plurality of statement scene images arranged in a preset order, wherein each text statement corresponds to at least one statement scene image; the image features of the virtual scene image include: a plurality of statement image features arranged in a preset order; wherein each statement image feature corresponds to one statement scene image.

[0228] In one optional implementation, the plurality of text statements arranged in a preset order are: a plurality of game text statements extracted from a plurality of game video frames of the game application according to a preset keyword list;

[0229] The preset order is determined based on the playback order of multiple game video frames corresponding to multiple game text statements; the preset keyword list is used to store game keywords corresponding to game types.

[0230] Furthermore, the scene image of the statement is determined based on the game video frame corresponding to the text statement.

[0231] In one alternative implementation, the fusion module is specifically adapted to:

[0232] For any text statement, based on the statement image features of at least one related statement scene image of the text statement, interactive fusion processing is performed on the statement text features of the text statement to obtain the statement fusion features of the text statement.

[0233] The statement fusion features of multiple text statements are arranged in the preset order to obtain the text fusion features of the initial text;

[0234] Wherein, at least one related statement scene image of any text statement includes: a target scene image containing the any text statement, and / or an associated scene image that is related to the any text statement.

[0235] In one alternative implementation, the fusion module is specifically adapted to:

[0236] For any text statement, a target scene image containing the text statement is determined. Based on the statement image features of the target scene image, intra-sentence fusion processing is performed on the statement text features of the text statement to obtain the intra-sentence fusion features of the text statement.

[0237] Determine the associated text statement corresponding to any of the text statements, and perform inter-sentence fusion processing on the intra-sentence fusion features of the associated text statements based on the intra-sentence fusion features of the associated text statements to obtain the statement fusion features of the any of the text statements.

[0238] In one alternative implementation, the fusion module is specifically adapted to:

[0239] Calculate the statement similarity between other text statements and any given text statement, and determine the text statements with a statement similarity greater than a preset similarity threshold as associated text statements corresponding to any given text statement; wherein, the statement similarity is determined based on the text similarity between other text statements and any given text statement, and the image similarity between the statement scene image corresponding to the other text statements and the statement scene image corresponding to the given text statement; or,

[0240] Determine the identification information corresponding to any of the text statements, and determine the associated text statements corresponding to any of the text statements based on the identification information; wherein, the identification information includes: game stage information, and / or chapter / paragraph information.

[0241] In one optional implementation, the text statement contains multiple text characters, and the statement text features of the text statement specifically include: multiple character features, each character feature corresponding to a text character;

[0242] The intra-sentence fusion features of any text statement specifically include: multiple intra-sentence character features, and each intra-sentence character feature corresponds to a text character;

[0243] The intra-sentence character features of each text character are calculated in the following way:

[0244] Calculate the character relevance between the character features of the text characters and the sentence image features of the target scene image, and set the character fusion weights based on the character relevance;

[0245] Based on the character fusion weight, the sentence image features of the target scene image are weighted and fused with the character features to obtain the intra-sentence character features of the text characters.

[0246] In one alternative implementation, the character characteristics of any text character in any text statement are obtained in the following way:

[0247] Obtain the initial character vector for each text character in the text statement;

[0248] For any text character in the text statement, the initial character vector of the any text character is subjected to intra-sentence interactive fusion processing with the initial character vectors of the other text characters in the text statement to obtain the fused character vector of the any text character; wherein, the fused character vector of the any text character is used to characterize the character features of the any text character.

[0249] In one optional implementation, the associated text statement corresponding to any one of the text statements includes: a first type of associated text statement whose statement similarity to any one of the text statements is greater than a preset first similarity threshold, and a second type of associated text statement whose statement similarity to any one of the text statements is not greater than the preset first similarity threshold.

[0250] The fusion module is specifically adapted to:

[0251] For any text character in any text statement, perform a first inter-sentence fusion process based on the first type of associated text statements to obtain the first inter-sentence character feature of the text character; perform a second inter-sentence fusion process based on the second type of associated text statements to obtain the second inter-sentence character feature of the text character.

[0252] Based on the character features between the first and second sentences, determine the character fusion features of any text character;

[0253] The statement fusion feature of any text statement is determined based on the character fusion features of multiple text characters in any text statement.

[0254] In one optional implementation, the fusion module is specifically adapted to: obtain the intra-sentence character features of a first number of text characters contained in the first type of associated text statement, and fuse the intra-sentence character features of the first number of text characters with the intra-sentence character features of any text character in any text statement to obtain the first inter-sentence character features of any text character;

[0255] Obtain the intra-sentence character features of a second number of text characters contained in the second type of associated text statement, and fuse the intra-sentence character features of the second number of text characters with the intra-sentence character features of any text character in any text statement to obtain the second inter-sentence character features of any text character; wherein, the first number is greater than the second number.

[0256] In one alternative implementation, the image acquisition module is specifically adapted to:

[0257] Acquire image data, interactive element data, and / or game stage data of the virtual scene image;

[0258] The image data, interactive element data, and / or game stage data are vectorized to obtain the image features of the virtual scene image.

[0259] In one alternative implementation, the prediction module is specifically adapted to:

[0260] Obtain the image features of subsequent scene images, and input the image features of the subsequent scene images into the text prediction model;

[0261] The text prediction model obtains the subsequent text predicted by the subsequent scene image based on the image features and the text fusion features of the initial text.

[0262] In one optional implementation, the virtual scene is a game scene in a game application, and the prediction module is specifically adapted to:

[0263] Display multiple candidate scene images;

[0264] In response to an image selection command triggered by a user for any candidate scene image, the image features of subsequent scene images are obtained based on the image identifier contained in the image selection command.

[0265] In one alternative implementation, the text prediction model is trained in the following manner:

[0266] Obtain initial samples in the virtual scene and determine the sample features corresponding to the initial samples;

[0267] Obtain a sample scene image in the virtual scene corresponding to the initial sample, and determine the image features of the sample scene image;

[0268] Based on the image features of the sample scene image, an interactive fusion process is performed on the sample features of the initial sample to obtain the sample fusion features of the initial sample;

[0269] The text prediction model is trained based on the sample fusion features of the initial samples and the pre-annotated label text of the initial samples.

[0270] In one alternative implementation, the fusion module is specifically adapted to:

[0271] The text features of the initial text are adjusted using the image features of the virtual scene image, and the adjusted text features are used as the text fusion features of the initial text.

[0272] Among them, the image features of the virtual scene image are image modal features, the text features of the initial text are text modal features, and the text fusion features of the initial text are multimodal features that fuse the image modal features and the text modal features.

[0273] The specific structure and working principle of each of the above modules can be found in the description of the corresponding part of the method embodiment, and will not be repeated here.

[0274] Another embodiment of this application provides a non-volatile computer storage medium storing at least one executable instruction that can execute the object loading method in the virtual scene of any of the above method embodiments. Specifically, the executable instruction can be used to cause the processor to perform the corresponding operations in the above method embodiments.

[0275] Figure 6 The diagram shows a structural schematic of an electronic device according to yet another embodiment of the present disclosure. The specific embodiments of the present disclosure do not limit the specific implementation of the electronic device.

[0276] like Figure 6 As shown, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0277] in:

[0278] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0279] Communication interface 504 is used to communicate with other network elements such as clients or other servers.

[0280] The processor 502 is used to execute program 510, specifically to execute the relevant steps in the above-described video image detection method embodiment.

[0281] Specifically, program 510 may include program code that includes computer operation instructions.

[0282] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure. The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0283] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0284] Specifically, program 510 can be used to cause processor 502 to perform the corresponding operations in the above-described video image detection method embodiment.

[0285] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is apparent from the above description. Furthermore, this disclosure is not directed to any particular programming language. It should be understood that the contents of this disclosure described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this disclosure.

[0286] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0287] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various aspects of the disclosure, in the above description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed disclosure requires more features than are expressly recited in each claim.

[0288] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature serving the same, equivalent, or similar purpose.

[0289] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this disclosure and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0290] The various component embodiments of this disclosure can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to embodiments of this disclosure. This disclosure can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such an implementation of this disclosure can be stored on a computer-readable medium or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0291] It should be noted that the above embodiments are illustrative of this disclosure and not restrictive, and that alternative embodiments can be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This disclosure can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A text prediction method based on a virtual scene, comprising: Obtain the initial text in the virtual scene and determine the text features corresponding to the initial text; Obtain the virtual scene image corresponding to the initial text in the virtual scene, and determine the image features of the virtual scene image; Based on the image features of the virtual scene image, interactive fusion processing is performed on the text features of the initial text to obtain the text fusion features of the initial text; wherein, the text features of the initial text are adjusted using the image features of the virtual scene image, and the adjusted text features are used as the text fusion features of the initial text; wherein, the image features of the virtual scene image are image modal features, and the text features of the initial text are text modal features; the text fusion features of the initial text are multimodal features that fuse the image modal features and the text modal features; Based on the text fusion features of the initial text, predict the subsequent text in the virtual scene. The subsequent text in the virtual scene refers to text content that has a semantic connection with the initial text in the virtual scene.

2. The method according to claim 1, wherein, The initial text in the virtual scene specifically includes: multiple text statements arranged in a preset order; the text features of the initial text specifically include: multiple statement text features arranged in a preset order, wherein each statement text feature corresponds to a text statement. Furthermore, the virtual scene image specifically includes: multiple statement scene images arranged in a preset order, wherein each text statement corresponds to at least one statement scene image; the image features of the virtual scene image specifically include: multiple statement image features arranged in a preset order; wherein each statement image feature corresponds to one statement scene image.

3. The method according to claim 2, wherein, The multiple text statements arranged in a preset order are: multiple game text statements extracted from multiple game video frames of the game application according to a preset keyword list; The preset order is determined based on the playback order of multiple game video frames corresponding to multiple game text statements; the preset keyword list is used to store game keywords corresponding to game types. Furthermore, the scene image of the statement is determined based on the game video frame corresponding to the text statement.

4. The method according to claim 2, wherein, The step of performing interactive fusion processing on the text features of the initial text based on the image features of the virtual scene image to obtain the text fusion features of the initial text specifically includes: For any text statement, based on the statement image features of at least one related statement scene image of the text statement, interactive fusion processing is performed on the statement text features of the text statement to obtain the statement fusion features of the text statement. The statement fusion features of multiple text statements are arranged in the preset order to obtain the text fusion features of the initial text; Wherein, at least one related statement scene image of any text statement includes: a target scene image containing the any text statement, and / or an associated scene image that is related to the any text statement.

5. The method according to claim 4, wherein, The step of performing interactive fusion processing on the sentence text features of any given text statement based on the sentence image features of at least one related sentence scene image of the given text statement to obtain the sentence fusion features of the given text statement specifically includes: For any text statement, a target scene image containing the text statement is determined. Based on the statement image features of the target scene image, intra-sentence fusion processing is performed on the statement text features of the text statement to obtain the intra-sentence fusion features of the text statement. Determine the associated text statement corresponding to any of the text statements, and perform inter-sentence fusion processing on the intra-sentence fusion features of the associated text statements based on the intra-sentence fusion features of the associated text statements to obtain the statement fusion features of the any of the text statements.

6. The method according to claim 5, wherein, The determination of the associated text statement corresponding to any text statement specifically includes: Calculate the statement similarity between other text statements and any given text statement, and determine the text statements with a statement similarity greater than a preset similarity threshold as associated text statements corresponding to any given text statement; wherein, the statement similarity is determined based on the text similarity between other text statements and any given text statement, and the image similarity between the statement scene image corresponding to the other text statements and the statement scene image corresponding to the given text statement; or, Determine the identification information corresponding to any of the text statements, and determine the associated text statements corresponding to any of the text statements based on the identification information; wherein, the identification information includes: game stage information, and / or chapter / paragraph information.

7. The method according to claim 5, wherein, The text statement contains multiple text characters, and the text features of the text statement specifically include: multiple character features, each character feature corresponding to a text character; The intra-sentence fusion features of any text statement specifically include: multiple intra-sentence character features, and each intra-sentence character feature corresponds to a text character; The intra-sentence character features of each text character are calculated in the following way: Calculate the character relevance between the character features of the text characters and the sentence image features of the target scene image, and set the character fusion weights based on the character relevance; Based on the character fusion weight, the sentence image features of the target scene image are weighted and fused with the character features to obtain the intra-sentence character features of the text characters.

8. The method according to claim 7, wherein, The character characteristics of any text character in any text statement are obtained in the following way: Obtain the initial character vector for each text character in the text statement; For any text character in the text statement, the initial character vector of the any text character is subjected to intra-sentence interactive fusion processing with the initial character vectors of the other text characters in the text statement to obtain the fused character vector of the any text character; wherein, the fused character vector of the any text character is used to characterize the character features of the any text character.

9. The method according to claim 7 or 8, wherein, The associated text statements corresponding to any one of the text statements specifically include: a first type of associated text statements whose statement similarity to any one of the text statements is greater than a preset first similarity threshold, and a second type of associated text statements whose statement similarity to any one of the text statements is not greater than the preset first similarity threshold; The step of performing inter-sentence fusion processing on the intra-sentence fusion features of any text statement based on the intra-sentence fusion features of the associated text statements to obtain the statement fusion features of any text statement specifically includes: For any text character in any text statement, perform a first inter-sentence fusion process based on the first type of associated text statements to obtain the first inter-sentence character feature of the text character; perform a second inter-sentence fusion process based on the second type of associated text statements to obtain the second inter-sentence character feature of the text character. Based on the character features between the first and second sentences, determine the character fusion features of any text character; The statement fusion feature of any text statement is determined based on the character fusion features of multiple text characters in any text statement.

10. The method according to claim 9, wherein, The step of performing a first inter-sentence fusion process based on the first type of associated text statements to obtain the first inter-sentence character feature of any text character specifically includes: obtaining the intra-sentence character features of a first number of text characters contained in the first type of associated text statements, and performing a fusion process between the intra-sentence character features of the first number of text characters and the intra-sentence character features of any text character in any text statement to obtain the first inter-sentence character feature of any text character; The step of performing a second inter-sentence fusion process based on the second type of associated text statements to obtain the second inter-sentence character features of any text character specifically includes: obtaining the intra-sentence character features of a second number of text characters contained in the second type of associated text statements, and fusing the intra-sentence character features of the second number of text characters with the intra-sentence character features of any text character in any text statement to obtain the second inter-sentence character features of any text character; wherein, the first number is greater than the second number.

11. The method according to any one of claims 1-8, wherein, The determination of the image features of the virtual scene image specifically includes: Acquire image data, interactive element data, and / or game stage data of the virtual scene image; The image data, interactive element data, and / or game stage data are vectorized to obtain the image features of the virtual scene image.

12. The method according to any one of claims 1-8, wherein, The step of predicting subsequent text in the virtual scene based on the text fusion features of the initial text specifically includes: Obtain the image features of subsequent scene images, and input the image features of the subsequent scene images into the text prediction model; The text prediction model obtains the subsequent text predicted by the subsequent scene image based on the image features and the text fusion features of the initial text.

13. The method according to claim 12, wherein, The virtual scene is a game scene in a game application, and the image features for obtaining subsequent scene images specifically include: Display multiple candidate scene images; In response to an image selection command triggered by a user for any candidate scene image, the image features of subsequent scene images are obtained based on the image identifier contained in the image selection command.

14. The method according to claim 12, wherein, The text prediction model is trained in the following way: Obtain initial samples in the virtual scene and determine the sample features corresponding to the initial samples; Obtain a sample scene image in the virtual scene corresponding to the initial sample, and determine the image features of the sample scene image; Based on the image features of the sample scene image, an interactive fusion process is performed on the sample features of the initial sample to obtain the sample fusion features of the initial sample; The text prediction model is trained based on the sample fusion features of the initial samples and the pre-annotated label text of the initial samples.

15. A text prediction device based on a virtual scene, comprising: The text acquisition module is adapted to acquire initial text in the virtual scene and determine text features corresponding to the initial text; An image acquisition module is adapted to acquire a virtual scene image corresponding to the initial text in the virtual scene and determine the image features of the virtual scene image; The fusion module is adapted to perform interactive fusion processing on the text features of the initial text based on the image features of the virtual scene image to obtain the text fusion features of the initial text; wherein, the text features of the initial text are adjusted using the image features of the virtual scene image, and the adjusted text features are used as the text fusion features of the initial text; wherein, the image features of the virtual scene image are image modal features, and the text features of the initial text are text modal features; the text fusion features of the initial text are multimodal features that fuse the image modal features and the text modal features; The prediction module is adapted to predict subsequent text in the virtual scene based on the text fusion features of the initial text. The subsequent text in the virtual scene refers to text content that has a semantic connection with the initial text in the virtual scene.

16. An electronic device comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the text prediction method as described in any one of claims 1-14.

17. A computer storage medium storing at least one executable instruction that causes a processor to perform the text prediction method as described in any one of claims 1-14.

Images