A method and system for automatically constructing four-dimensional artistic conception based on artificial intelligence context

Through the four-dimensional artistic conception automatic construction method based on the context of artificial intelligence, three-dimensional scenes related to ancient poetry and realize four-dimensional time series, the problems of existing AI drawing models in the generation and uncertainty of two-dimensional scenes are solved, and more intuitive Ancient poetry experience.

CN118470160BActive Publication Date: 2024-12-03XIAMEN YIXUE SOFTWARE CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410925507.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-12-03
Estimated Expiration
2044-07-11

AI Technical Summary

Technical Problem

The existing AI drawing models are limited to two-dimensional scene generation, and the generation results are highly uncertain, so they cannot effectively construct a four-dimensional artistic conception, making it difficult for users to intuitively feel the content and connotation of ancient poems.

Method used

Adopt the four-dimensional artistic conception automatic construction method based on the artificial intelligence context, and receive ancient poems and sentences, extract word segmentation, use the three-dimensional material library and attribute library to generate three-dimensional artistic conception, and drive actions through tensor smoothing technology to generate four-dimensional time series to realize four-dimensional rendering .

Benefits of technology

The three-dimensional scene generation and dynamic changes of ancient poetry content are realized, and users can feel the artistic conception and connotation of poetry more intuitively, improving the generation efficiency and the closeness of results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470160B_ABST
    Figure CN118470160B_ABST
Patent Text Reader

Abstract

A method and system for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context, which relates to the teaching application field of information technology, and the method includes: receiving input ancient poetry sentences, extracting sentence segmentation using a trained word library model; for nouns, extracting corresponding materials from an established three-dimensional material library; for adjectives, modifying the characteristics of the materials based on an established attribute library; using a trained artificial intelligence rendering engine, arranging the processed materials in space to obtain a three-dimensional artistic conception; for verbs, adding corresponding actions of the materials based on an established action library; using tensor smoothing technology, driving the materials in the three-dimensional artistic conception through actions, and smoothing the intermediate process of the actions to generate a four-dimensional time series to achieve four-dimensional rendering; and outputting the four-dimensional artistic conception after four-dimensional rendering. The present invention can automatically analyze the content of ancient poetry, generate a four-dimensional artistic conception related to the content, and enable users to more intuitively and immersively experience the content, artistic conception and connotation of ancient poetry.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of teaching application of information technology, and in particular to a method and system for automatically constructing four-dimensional artistic conception based on artificial intelligence context. Background Art

[0002] In recent years, AI drawing models represented by Stable Fusion and Midjourney have received great attention. They can generate images, videos and other materials based on user input of keywords, sample images and other information. However, such models are limited to two-dimensional scenes, and the generated results are highly uncertain.

[0003] For example, the Chinese invention patent application "A system for intelligently generating ancient poetry context videos by AI and its working method (application number 2022113986085) constructs an AI intelligent system for generating ancient poetry context videos. The system automatically generates context videos corresponding to the content based on the ancient poetry text, which can provide a large amount of video learning resources for teachers' teaching and students' self-study activities, reduce the difficulty of understanding, provide a new way to generate ancient poetry learning videos, and promote the intelligent generation of ancient poetry teaching resources". On the one hand, the generation process of the ancient poetry context videos in this application is completely dependent on stable diffusion, which is somewhat uncontrollable; on the other hand, the output of this application is a two-dimensional video.

[0004] The Chinese invention patent application "Scene construction method, device, equipment and storage medium" (application number 2022109165555) discloses "extracting at least one scene keyword from the received text information; for at least one scene keyword, obtaining scene parameters matching the scene keyword from a preset scene parameter library; determining a three-dimensional material matching the scene keyword based on the scene parameters; and constructing a three-dimensional scene matching the text information based on at least one of the three-dimensional materials." That is, it discloses that a three-dimensional scene is automatically constructed by extracting scene keywords from text information, thereby realizing the automatic construction of a three-dimensional scene. However, four-dimensional time is not involved in constructing the scene, and a static three-dimensional scene is output. Summary of the invention

[0005] The main purpose of the present invention is to propose a method and system for automatically constructing four-dimensional artistic conception based on artificial intelligence context, which can automatically analyze the content of ancient poetry, generate three-dimensional artistic conception related to the content, and further generate four-dimensional time series through the movement of scene objects, so that users can more intuitively and immersively experience the content, artistic conception and connotation of ancient poetry.

[0006] The present invention adopts the following technical solution:

[0007] On the one hand, a method for automatically constructing four-dimensional artistic conception based on artificial intelligence context includes:

[0008] Receive input ancient poetry sentences, and use the trained word library model to extract sentence segmentation; the word segmentation includes noun, adjective and verb;

[0009] For nouns, corresponding materials are extracted from the established three-dimensional material library;

[0010] For adjectives, modify the characteristics of the material based on the established attribute library;

[0011] Use the trained AI rendering engine to arrange the processed materials in space to obtain a three-dimensional concept including the size, position, and front-to-back relationship of the materials in the image;

[0012] For verbs, add corresponding actions of the material based on the established action library;

[0013] Utilizing tensor smoothing technology, the material in the three-dimensional artistic conception is driven by action, and the intermediate process of the action is smoothed to generate a four-dimensional time series to achieve four-dimensional rendering;

[0014] Output the four-dimensional artistic conception after four-dimensional rendering;

[0015] Using tensor smoothing technology, the material in the three-dimensional artistic conception is driven by action, and the intermediate process of the action is smoothed to generate a four-dimensional time series to achieve four-dimensional rendering, including:

[0016] Use the size, position and front-to-back relationship of the material in the image to place the three-dimensional material, and use the actions in the action library to drive the movement of the material; after adding the action, a four-dimensional tensor V (X, Y, Z, T) is generated to represent the four-dimensional artistic conception to reflect the change of the three-dimensional artistic conception over time; for multi-channel images, a feature dimension F is added;

[0017] Three-dimensional artistic conception at any time V t It is expressed as follows:

[0018]

[0019] Among them, R1, R2 and R3 represent the number of low-rank components after decomposition of the four dimensions of three-dimensional space plus time; and Represent the low-rank components respectively; Represents a pixel matrix composed in the XY plane; Represents the pixel matrix composed in the XZ plane; Represents the pixel matrix composed in the ZY plane; Indicates V t The vector along the Z axis; Indicates V t The vector along the Y axis; Indicates V t The vector along the X axis; Indicates V t A vector of dimension F along the XY plane; Indicates V t A vector of dimension F along the XZ plane; Indicates V t The vector of dimension F along the YZ plane; f k (t) r Represents the weight of the corresponding low-rank component at time point t, k∈[1,3];

[0020] The function f k (t) r Set it as the MLP function in the neural network, and use the MLP function to Mapping is performed to obtain the feature D after processing by the neural network, as follows:

[0021]

[0022] in, Represents the pixel matrix composed in the ZT plane; Represents the pixel matrix composed in the YT plane; Represents the pixel matrix composed in the XT plane;

[0023] The pixels on the plane are stacked to obtain the feature representation D(x, y, z, t) of any pixel point, as follows:

[0024]

[0025]

[0026] in, express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stack of all pixels in ; express The stack of all pixels in ; express The stack of all pixels in ;

[0027] Then and Stack to V RF , get the feature representation of any pixel as follows:

[0028]

[0029] Among them, the semicolon means connecting all the features;

[0030] The loss function is minimized for the adjacent pixels on the six surfaces to obtain the smoothed four-dimensional artistic conception pixel points.

[0031] Preferably, the input of the vocabulary model is ancient poetry sentences;

[0032] The output of the vocabulary model is a vocabulary table, the fields of which include original words, replacement words, word frequency and part of speech; the original words are original participles extracted from ancient poems and sentences; the replacement words are participles that replace the original words when the word frequency of the original words is lower than a preset value.

[0033] Preferably, the vocabulary model is an improved Unet model, and the model training process is as follows:

[0034] Fill the word frequency distribution map of ancient poetry sentences with zeros to 24x24;

[0035] Create a 3x3 first convolutional layer and a second convolutional layer with 4-channel outputs; the input of the second convolutional layer comes from the output of the first convolutional layer;

[0036] After the first convolutional layer and the second convolutional layer are connected in series, the output is sent to a third convolutional layer with a single channel output of 3x3;

[0037] Use the loss function logistic=t_value for training to obtain the vocabulary table.

[0038] Preferably, in the three-dimensional material library, the materials are classified and described based on the classification, and the description includes: description of application field, description of environment, description of objects and description of characters.

[0039] Preferably, the action library describes the action process for use in action processing in four-dimensional rendering.

[0040] Preferably, the input of the artificial intelligence rendering engine is an image, and the image includes at least one material; the output of the artificial intelligence rendering engine includes the size, position and front-back relationship of the material in the image;

[0041] The training process of the AI ​​rendering engine is as follows:

[0042] Selecting an image including at least one material as a training set; a segmentation data mark is set in the image;

[0043] The segmentation data markers in the image are used to train the artificial intelligence rendering engine to output the size, position and front-back relationship of different materials in the image; wherein the segmentation data markers include the pixel category, size, position and front-back relationship contained in the object in the image, and the pixel category corresponds to the material category in the three-dimensional material library.

[0044] Preferably, in the minimization of the loss functions for the adjacent pixels on the six planes respectively, the corresponding loss functions are as follows:

[0045]

[0046] Where i and j represent the pixel numbers; β is a preset constant; P represents or

[0047] On the other hand, a four-dimensional artistic conception automatic construction system based on artificial intelligence context includes:

[0048] A word segmentation module is used to receive an input ancient poetry sentence and extract the sentence segmentation using a trained word library model; the part of speech of the segmentation includes noun, adjective and verb;

[0049] The material extraction module, for nouns, is used to extract corresponding materials from the established three-dimensional material library;

[0050] The attribute modification module, for adjectives, is used to modify the characteristics of the material based on the established attribute library;

[0051] The 3D artistic conception generation module is used to use the trained artificial intelligence rendering engine to arrange the processed materials in space to obtain a 3D artistic conception;

[0052] The action adding module is used to add corresponding actions of materials based on the established action library for verbs;

[0053] The four-dimensional rendering module is used to use tensor smoothing technology to drive the material in the three-dimensional artistic conception through action, and smooth the intermediate process of the action to generate a four-dimensional time series to achieve four-dimensional rendering;

[0054] The output module is used to output the four-dimensional artistic conception after four-dimensional rendering.

[0055] In a third aspect, a computer device includes a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor implements the step of automatically constructing a four-dimensional artistic conception based on an artificial intelligence context when executing the computer program.

[0056] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The present invention can automatically analyze the content of ancient poems and generate three-dimensional scenes related to the content. Compared with the existing two-dimensional video generation method, the user can more intuitively and immersively experience the content, artistic conception and connotation of the ancient poems;

[0059] (2) The present invention can automatically generate descriptions of materials faster by accumulating a material library, and can also automatically edit the materials in the scene using the contents of the attribute library and the action library, thereby improving the generation efficiency and making the generated results closer to the content of the ancient poems;

[0060] (3) The present invention utilizes image segmentation technology and deep learning technology to determine the size and position of the material in the three-dimensional artistic conception, thereby further improving the degree of automation of artistic conception generation.

[0061] (4) The present invention utilizes tensor smoothing technology to drive the material in the artistic conception through action and smooth the intermediate process of the action, thereby realizing the generation of a four-dimensional time series, so that the changing three-dimensional artistic conception in the poem can be dynamically presented over time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of a method and system for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context according to an embodiment of the present invention;

[0063] Figure 2 It is a structural schematic diagram of an improved Unet model according to an embodiment of the present invention;

[0064] FIG3(a) is an input image of an artificial intelligence rendering engine according to an embodiment of the present invention; FIG3(b) is an output image of an artificial intelligence rendering engine according to an embodiment of the present invention;

[0065] Figure 4 It is a structural schematic diagram of a method and system for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context according to an embodiment of the present invention;

[0066] Figure 5Schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0068] See also Figure 1 As shown, this embodiment provides a method and system for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context, including:

[0069] S101, receiving an input ancient poetry sentence, and extracting sentence segmentation using a trained word library model; the part of speech of the segmentation includes noun, adjective and verb;

[0070] S102, for the noun, extracting corresponding materials from an established three-dimensional material library;

[0071] S103, for adjectives, modifying the characteristics of the material based on the established attribute library;

[0072] S104, using a trained artificial intelligence rendering engine to arrange the processed material in space to obtain a three-dimensional image including the size, position, and front-back relationship of the material in the image;

[0073] S105, for the verb, adding the corresponding action of the material based on the established action library;

[0074] S106, using tensor smoothing technology, driving the material in the three-dimensional artistic conception through action, and smoothing the intermediate process of the action to generate a four-dimensional time series, thereby realizing four-dimensional rendering;

[0075] S107, outputting the four-dimensional artistic conception after four-dimensional rendering.

[0076] In this embodiment, the received input ancient poetry sentence can be an ancient poetry sentence input in text form, or an ancient poetry sentence input in voice form. The specific setting is based on the needs and this embodiment does not impose any limitation.

[0077] In addition, for the input ancient poetry sentences, the corresponding four-dimensional artistic conception can be output after receiving a sentence, or after receiving a paragraph, or after receiving the whole poem. The specific setting is based on needs, and this embodiment does not limit it.

[0078] In this embodiment, the processing process of the vocabulary model is as follows.

[0079] Generally speaking, the words that appear in ancient poetry sentences are relatively short, usually one or two words, so this embodiment establishes a one-word and two-word word library.

[0080] Specifically, the input of the vocabulary model is ancient poetry sentences; the output of the vocabulary model is a vocabulary table, and the fields of the vocabulary table include original words, replacement words, word frequency and part of speech; the original words are original participles extracted from ancient poetry sentences; the replacement words are participles that replace the original words when the word frequency of the original words is lower than a preset value.

[0081] See also Figure 2 As shown, the vocabulary model is an improved Unet model, and the model training process is as follows:

[0082] 1) Fill the word frequency distribution map of ancient poetry sentences with zeros to 24x24 (inputLayer);

[0083] 2) Establish the first convolutional layer and the second convolutional layer with 3x3 outputs of 4 channels. The first convolutional layer is represented as (Conv1, bn_Conv1, Conv1_relu), and the second convolutional layer is represented as (Conv2, bn_Conv2, Conv2_relu), where the input of the second convolutional layer comes from the first convolutional layer;

[0084] 3) Connect the two convolutional layers in series (Concaten1), perform another 3x3 convolution (Conv3), and output a single channel;

[0085] 4) Use the loss function logistic=t_value for training to obtain the vocabulary model of the training number.

[0086] In the vocabulary table output by the vocabulary model, words that are difficult to present, overly common pronouns, and proper nouns with low word frequency are deleted and replaced with common nouns such as "person" and "building", and the vocabulary attributes are manually marked; such as verbs, nouns, adjectives, etc. For example: The sun sets behind the mountains. In the vocabulary extraction, four words, namely, day, by, mountain, and end, can be obtained, which are manually marked as noun, verb, noun, and verb respectively.

[0087] In this embodiment, the method for establishing the three-dimensional material library is as follows.

[0088] Build a 3D material library for the nouns in the vocabulary. Initially define and classify the scene materials, and describe their categories and properties, mainly including:

[0089] 1) Description of application areas: such as indoor, outdoor, etc.;

[0090] 2) Description of the environment: such as forests, mountains, etc.;

[0091] 3) Description of items: such as ships, buildings, etc.;

[0092] 4) Characters: Characters with characteristics of different dynasties, such as the Tang Dynasty, the Song Dynasty, etc.

[0093] In this embodiment, the method for establishing the attribute library is as follows.

[0094] Construct an attribute library for the adjectives in the thesaurus.

[0095] The attribute library is associated with the content of the "material library". For example, from the analysis of "The sun along the mountain bows; Yellow river seawards flows", it can be seen that the "sun" has the attribute of "white", so "white" in the attribute library can be used as an attribute of the "sun". And from the analysis of "When spring comes, the river's waves, in crimson dressed, outshine the fire; In green as blue the river in spring does glide", it can be seen that "red" can also be used as an attribute of the "sun".

[0096] In this embodiment, the method for establishing the action library is as follows.

[0097] Construct an action library for the verbs in the thesaurus.

[0098] Describe the action process in the scene for action processing by 3D generation software.

[0099] For example: for the action of "walking", move the target forward and backward; for the action of "riding", place the target on a means of transportation.

[0100] It should be noted that the processing of the thesaurus can also use "Jia Yan" as a framework to realize the construction of the thesaurus, word segmentation, part-of-speech tagging, and short sentences. The final result is several sentences of an ancient poem and the keywords (subject-predicate-object, attributive, adverbial, complement) that explain the artistic conception in each sentence.

[0101] Specifically, the input of the artificial intelligence rendering engine is an image, and the image includes at least one kind of material; the output of the artificial intelligence rendering engine includes the size, position, and front-back relationship of the materials in the image.

[0102] The training process of the artificial intelligence rendering engine is as follows:

[0103] (1) Select an image including at least one kind of material as the training set; segmentation data marks are set in the image, and the segmentation data marks include the pixel category, size, position, and front-back relationship of the objects in the image, where the pixel category corresponds to the material category in the 3D material library;

[0104] (2) During training, simulate the segmentation process, randomly select a small window from the pixels of the segmented target for prediction, and compare the prediction result with the correct mark. The loss function for comparison is:

[0105] L = L fl +Ldice

[0106] in:

[0107] The first loss function L fl It is expressed as follows:

[0108] L fl =-(1-p t ) γ log(p t )

[0109]

[0110] Among them, p t It reflects the closeness between the classification of pixels after image segmentation and the correct segmentation answer, i.e., category y; γ>0 is an adjustable factor; y represents the correct image segmentation answer; Represents the result of the softmax output of the neural network. Definition of the Softmax function:

[0111]

[0112] where z i is the output value of the ith node, and C is the number of output nodes, that is, the number of pixel classification categories after image segmentation. The Softmax function can be used to convert the output values ​​of multiple classifications into a probability distribution ranging from [0, 1] and 1.

[0113]

[0114] The second loss function L dice It is expressed as follows:

[0115]

[0116] Among them, |X| and |Y| represent the number of pixels in the two sets X and Y; |X| is the prediction result, and |Y| is the correct label.

[0117] The simulation segmentation process is as follows:

[0118] (a) Image encoding: We use a pre-trained visual Transformer from the Masked AutoEncoders paper, minimally adapted to high-resolution inputs. The encoder is run only once per image before prompt encoding.

[0119] Input an image of (c, h, w), scale the image, scale the long side to 1024, and fill the short side if it is not enough, and get an image of (c, 1024, 1024). After graphic encoding, get the features of the image downsampled 16 times, and the size is (256, 64, 64);

[0120] (b) Hint encoding: Mapping a pixel point to a 256-dimensional vector, which contains the spatial position and the front and back position relationship of the point;

[0121] (c) Mask decoding: First, use self-attention extraction on the result of hint encoding to obtain the self-attention feature of hint encoding; secondly, use this feature and the image feature to perform cross-attention to extract the cross-self-attention feature; then, use MLP to update the cross-self-attention feature; finally, use this feature and the image feature to perform cross-attention extraction again.

[0122] Repeat the above steps twice, connect them through the residual, and finally output the segmentation result (expressed in the form of a mask, that is, which object each pixel belongs to).

[0123] (d) Count the pixels belonging to the same object and calculate the size and position information of the object.

[0124] The segmentation data tags in the image are used to train the artificial intelligence rendering engine. After obtaining the different materials contained in the poem based on the text analysis results, the network rendering model can be used to output the size, position and front-to-back relationship of different materials in the image.

[0125] In this embodiment, the training process of the artificial intelligence rendering engine is as follows.

[0126] 1) Select images of people, animals, and scenery from VOC, ImageNet, COCO and other datasets as training sets;

[0127] 2) Use the data labels related to "segmentation" to train the neural network. The input of the neural network is the image, as shown in Figure 3(a); the output of the neural network is the size, position and front-back relationship of different targets in the image, as shown in Figure 3(b).

[0128] There are regions such as people and mountains and corresponding segmentation annotations. The segmentation annotation information is generally the region category corresponding to each pixel. Through the above training, it can help the generated scene determine the location, size and other information of different materials that should usually appear in the image.

[0129] Using tensor smoothing technology, the material in the three-dimensional artistic conception is driven by action, and the intermediate process of the action is smoothed to generate a four-dimensional time series to achieve four-dimensional rendering, including:

[0130] Use the size, position and front-to-back relationship of the material in the image to place the three-dimensional material, and use the action in the action library to drive the movement of the material; after adding the action, a four-dimensional tensor V (X, Y, Z, T) is generated to represent the four-dimensional artistic conception to reflect the change of the three-dimensional artistic conception over time; for multi-channel images (such as RGB), a feature dimension F is added;

[0131] Three-dimensional artistic conception at any time V t It is expressed as follows:

[0132]

[0133] Among them, R1, R2 and R3 represent the number of low-rank components after decomposition of the four dimensions of three-dimensional space plus time; and Represent the low-rank components respectively; Represents a pixel matrix composed in the XY plane; Represents the pixel matrix composed in the XZ plane; Represents the pixel matrix composed in the ZY plane; Indicates V t The vector along the Z axis; Indicates V t The vector along the Y axis; Indicates V t The vector along the X axis; Indicates V t A vector of dimension F along the XY plane; Indicates V t A vector of dimension F along the XZ plane; Indicates V t The vector of dimension F along the YZ plane; f k (t) r Represents the weight of the corresponding low-rank component at time point t, k∈[1,3].

[0134] The function f k (t) r Let MLP be the function in the neural network. The definition of MLP function is:

[0135] MLP(x)=W1(ReLU(W2x+b1))+b2

[0136] Among them, W1, W2, b1 and b2 are all parameters of the neural network, and ReLU is a commonly used activation function of the neural network.

[0137] ReLU(x)=max(0,x)

[0138] Use the MLP function to Mapping is performed to obtain the feature D after processing by the neural network, as follows:

[0139]

[0140] in, Represents the pixel matrix composed in the ZT plane; Represents the pixel matrix composed in the YT plane; Represents the pixel matrix composed in the XT plane;

[0141] The pixels on the plane are stacked to obtain the feature representation D(x, y, z, t) of any pixel point, as follows:

[0142]

[0143] in, express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stack of all pixels in ; express The stack of all pixels in ; express The stack of all pixels in ;

[0144] Then and Stack to V RF , get the feature representation of any pixel as follows:

[0145] The semicolon indicates that all features are connected.

[0146] The loss function is minimized for the adjacent pixels on the six faces to obtain the smoothed four-dimensional artistic conception pixel points. The loss function is as follows:

[0147]

[0148] Among them, i and j represent the pixel numbers; β is a preset constant; P represents or For example, based on the above formula, the difference between adjacent pixel points stacked on the XY plane can be obtained as:

[0149]

[0150] The following will be explained using two ancient poems as examples. It should be noted that this embodiment only processes the body sentences but not the titles.

[0151] Poem 1: Climbing the Stork Tower

[0152] The sun sets over the mountains

[0153] The Yellow River flows into the sea

[0154] To see the world from afar

[0155] Going to the next level

[0156] Poem 2: To Wang Lun

[0157] Li Bai is about to set off on a boat

[0158] Suddenly I heard singing on the shore

[0159] Taohuatan is a thousand feet deep

[0160] Not as good as Wang Lun's love for me

[0161] The processing steps of this embodiment are as follows.

[0162] Step 1: Create a vocabulary

[0163] Through training, the following segmentations can be extracted:

[0164] white, adjective

[0165] day, noun

[0166] Yi, verb

[0167] mountain, noun

[0168] to do, verb

[0169] yellow, adjective

[0170] river, noun

[0171] enter, verb

[0172] sea, noun

[0173] flow, verb

[0174] Desire, verb

[0175] Thousand miles, adjective

[0176] Head, noun

[0177] more, adjective

[0178] Up, verb

[0179] layer, adjective

[0180] Building, noun

[0181] Li Bai, noun

[0182] multiply, verb

[0183] boat, noun

[0184] will, adjective

[0185] line, verb

[0186] suddenly heard, verb

[0187] shore, noun

[0188] singing, noun

[0189] Peach Blossom Pool, noun

[0190] deep, verb

[0191] Thousand feet, adjective

[0192] less than, verb

[0193] Wang Lun, noun

[0194] send, verb

[0195] I, noun

[0196] emotion, noun

[0197] Among them, for proper nouns with lower word frequency, such as "Li Bai", "Wang Lun", "I" is replaced by "person", "Peach Blossom Pool" is replaced by "lake", and "stepping singing sound" is replaced by "sound".

[0198] Words that are difficult to present, such as qianchi, yuqiong, buqi, send, hear, and one layer, are deleted and not processed when constructing the scene.

[0199] Step 2: Build a material library for the nouns in the vocabulary.

[0200] The nouns that appear in the above two poems include "sun, mountain, river, sea, building, person, boat, lake", etc., and three-dimensional materials are created for each of them and added to the material library.

[0201] Step 3: Build an attribute library for the adjectives in the vocabulary.

[0202] The adjectives that appear in the above two poems are "white, yellow", etc. An attribute library for the materials is established. For example, for "white", the "sun" can be replaced with white, and for "yellow", the river water can be replaced with yellow.

[0203] Step 4: Train the rendering model.

[0204] By training the artificial intelligence rendering engine, the positions and sizes of "sun, mountain, river, sea, building, person, boat, lake" etc. that should appear in the image can be obtained.

[0205] The above is the training and preparation process, and the following is the construction process.

[0206] Read each line of the ancient poem in a loop. For example, first read "The sun along the mountain bows".

[0207] Step 5: Material extraction.

[0208] First, extract the materials corresponding to the nouns, such as "sun, mountain", etc.; then, modify the materials with attributes, change the sun to white; then read the actions. "Yi" means approaching, indicating that the "sun" and the "mountain" are close together, and "jin" means falling, indicating the fall of the "sun".

[0209] Step 6: Generation of three-dimensional artistic conception

[0210] Utilize the artificial intelligence rendering engine obtained from the training in Step 4 to render the corresponding objects from the background to the foreground according to the positional relationship.

[0211] For example: The sun along the mountain bows

[0212] 1) First, render the background objects such as the sky and mountains. In the above sentence, first render the "mountain";

[0213] 2) Secondly, render the foreground objects such as the sun, moon, houses, people, etc. In the above sentence, render the "white sun";

[0214] 3) The size of the objects is determined according to the training results of the artificial intelligence rendering engine.

[0215] After completion, read the next sentence, then render and switch to the next scene until all the poems are processed.

[0216] Step 7: Build an action library for the verbs in the thesaurus.

[0217] The verbs that appear in the above two poems are "yi, jin, liu, shang, cheng, shen", etc. Build an action library for the materials respectively. For example, "liu" means the flow of water, and "shen" means the camera moving underwater.

[0218] Step 8: Use the tensor smoothing technology to drive the materials in the three-dimensional artistic conception through actions, and smooth the intermediate process of the actions to generate a four-dimensional time series, realizing four-dimensional rendering and output.

[0219] For example, when the sun sets behind the mountains, "set" refers to the process of the sun setting. During this process of setting, the movement of the sun will use tensor smoothing technology, so that the sun will fall over time.

[0220] See also Figure 4 As shown, according to another aspect of the present invention, this embodiment further discloses a four-dimensional artistic conception automatic construction system based on artificial intelligence context, including:

[0221] The word segmentation module 401 is used to receive the input ancient poetry sentence and extract the sentence segmentation using the trained word library model; the part of speech of the segmentation includes noun, adjective and verb;

[0222] The material extraction module 402 is used to extract corresponding materials from the established three-dimensional material library for nouns;

[0223] The attribute modification module 403 is used to modify the characteristics of the material based on the established attribute library for adjectives;

[0224] The three-dimensional artistic conception generation module 404 is used to use the trained artificial intelligence rendering engine to arrange the processed materials in space to obtain a three-dimensional artistic conception;

[0225] The action adding module 405 is used for adding corresponding actions of materials based on the established action library for verbs;

[0226] A four-dimensional rendering module 406 is used to use tensor smoothing technology to drive the material in the three-dimensional artistic conception through action, and smooth the intermediate process of the action to generate a four-dimensional time series to achieve four-dimensional rendering;

[0227] The output module 407 is used to output the four-dimensional artistic conception after four-dimensional rendering.

[0228] A specific implementation of a system for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context. A method for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context is described in detail in this embodiment.

[0229] See also Figure 5 As shown, the computer device of this embodiment includes: a memory 501, a processor 502, a communication interface 503 and a bus 504. The memory 501, the processor 502 and the communication interface 503 are connected to each other through the bus 504.

[0230] The memory 501 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 501 may store a program. When the program stored in the memory 501 is executed by the processor 502, the processor 502 is used to execute the various steps of the method for automatically constructing a four-dimensional artistic conception based on an artificial intelligence context shown in the above embodiment.

[0231] The processor 502 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs to implement the various methods shown in the embodiments of the present application.

[0232] The processor 502 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the method of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 502 or by instructions in the form of software.

[0233] The processor 502 may also be a general purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0234] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501, and completes the functions required to be performed by the units included in the device of the present application in combination with its hardware.

[0235] The communication interface 503 may use, but is not limited to, a transceiver or other transceiver device to implement communication between the computer device and other devices or a communication network.

[0236] The bus 504 may include a path for transmitting information between various components of the computer device (eg, the memory 501 , the processor 502 , and the communication interface 503 ).

[0237] It should be understood that the computer device shown in the embodiment of the present application may be an electronic device, or may be a chip configured in an electronic device.

[0238] It should be noted that the data processing method and device in this application can be used in the field of database technology, and can also be used in any field other than the field of database technology. This application does not limit the application field of the data processing method and device.

[0239] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0240] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0241] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0242] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0243] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0244] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0245] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0246] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0247] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0248] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for automatically constructing four-dimensional artistic conception based on artificial intelligence context, characterized in that: include: Receive the input ancient poetry sentences and use the trained vocabulary model to extract sentence segmentation; The parts of speech of the participle include noun, adjective and verb; For nouns, corresponding materials are extracted from the established three-dimensional material library; For adjectives, modify the characteristics of the material based on the established attribute library; Use the trained AI rendering engine to arrange the processed materials in space to obtain a three-dimensional concept including the size, position, and front-to-back relationship of the materials in the image; For verbs, add corresponding actions of the material based on the established action library; Utilizing tensor smoothing technology, the material in the three-dimensional artistic conception is driven by action, and the intermediate process of the action is smoothed to generate a four-dimensional time series to achieve four-dimensional rendering; Output the four-dimensional artistic conception after four-dimensional rendering; Using tensor smoothing technology, the material in the three-dimensional artistic conception is driven by action, and the intermediate process of the action is smoothed to generate a four-dimensional time series to achieve four-dimensional rendering, including: Use the size, position and front-to-back relationship of the material in the image to place the three-dimensional material, and use the actions in the action library to drive the movement of the material; after adding the action, a four-dimensional tensor V (X, Y, Z, T) is generated to represent the four-dimensional artistic conception to reflect the change of the three-dimensional artistic conception over time. For multi-channel images, a feature dimension F is added; Three-dimensional artistic conception at any time V t It is expressed as follows: Among them, R1, R2 and R3 represent the number of low-rank components after decomposition of the four dimensions of three-dimensional space plus time; and Represent the low-rank components respectively; Represents a pixel matrix composed in the XY plane; Represents the pixel matrix composed in the XZ plane; Represents the pixel matrix composed in the ZY plane; Indicates V t The vector along the Z axis; Indicates V t The vector along the Y axis; Indicates V t The vector along the X axis; Indicates V t A vector of dimension F along the XY plane; Indicates V t A vector of dimension F along the XZ plane; Indicates V t The vector of dimension F along the YZ plane; f k (t) r Represents the weight of the corresponding low-rank component at time point t, k∈[1,3]; Use the MLP function to Map them separately to obtain the feature D after neural network processing, as follows: in, Represents the pixel matrix composed in the ZT plane; Represents the pixel matrix composed in the YT plane; Represents the pixel matrix composed in the XT plane; The pixels on the plane are stacked to obtain the feature representation D(x, y, z, t) of any pixel point, as follows: in, express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stack of all pixels in ; express The stack of all pixels in ; express The stack of all pixels in ; Then and Stack to V RF , get the feature representation of any pixel as follows: Among them, the semicolon means connecting all the features; The loss function is minimized for the adjacent pixels on the six surfaces to obtain the smoothed four-dimensional artistic conception pixel points.

2. The method for automatically constructing a four-dimensional artistic conception based on artificial intelligence context according to claim 1, characterized in that: The input of the vocabulary model is ancient poetry sentences; The output of the vocabulary model is a vocabulary table, the fields of which include original words, replacement words, word frequency and part of speech; the original words are original participles extracted from ancient poems and sentences; the replacement words are participles that replace the original words when the word frequency of the original words is lower than a preset value.

3. The method for automatically constructing a four-dimensional artistic conception based on artificial intelligence context according to claim 2, characterized in that: The vocabulary model is an improved Unet model, and the model training process is as follows: Fill the word frequency distribution map of ancient poetry sentences with zeros to 24x24; Create a 3x3 first convolutional layer and a second convolutional layer with 4-channel outputs. The input of the second convolutional layer comes from the output of the first convolutional layer. After the first convolutional layer and the second convolutional layer are connected in series, the output is sent to a third convolutional layer with a single channel output of 3x3; Use the loss function logistic=t_value for training to obtain the vocabulary table.

4. The method for automatically constructing a four-dimensional artistic conception based on artificial intelligence context according to claim 1, characterized in that: In the three-dimensional material library, materials are classified and described based on the classification, and the description includes: description of application field, description of environment, description of objects and description of characters.

5. The method for automatically constructing four-dimensional artistic conception based on artificial intelligence context according to claim 1, characterized in that: The action library describes the action process and is used for action processing in four-dimensional rendering.

6. The method for automatically constructing a four-dimensional artistic conception based on artificial intelligence context according to claim 1, characterized in that: The input of the artificial intelligence rendering engine is an image, and the image includes at least one material; the output of the artificial intelligence rendering engine includes the size, position and front-back relationship of the material in the image; The training process of the AI ​​rendering engine is as follows: Selecting an image including at least one material as a training set; a segmentation data mark is set in the image; The segmentation data markers in the image are used to train the artificial intelligence rendering engine to output the size, position and front-back relationship of different materials in the image; wherein the segmentation data markers include the pixel category, size, position and front-back relationship contained in the object in the image, and the pixel category corresponds to the material category in the three-dimensional material library.

7. The method for automatically constructing four-dimensional artistic conception based on artificial intelligence context according to claim 1, characterized in that: The loss functions are used to minimize the adjacent pixels on the six surfaces respectively, and the corresponding loss functions are as follows: Among them, i and j represent the pixel numbers; β is a preset constant; P represents or 8. A four-dimensional artistic conception automatic construction system based on artificial intelligence context, characterized in that: include: The word segmentation module is used to receive the input ancient poetry sentences and extract the sentence segmentation using the trained word library model; The parts of speech of the participle include noun, adjective and verb; The material extraction module, for nouns, is used to extract corresponding materials from the established three-dimensional material library; The attribute modification module, for adjectives, is used to modify the characteristics of the material based on the established attribute library; The 3D artistic conception generation module is used to use the trained artificial intelligence rendering engine to arrange the processed materials in space to obtain a 3D artistic conception; The action adding module is used to add corresponding actions of materials based on the established action library for verbs; The four-dimensional rendering module is used to use tensor smoothing technology to drive the material in the three-dimensional artistic conception through action, and smooth the intermediate process of the action to generate a four-dimensional time series to achieve four-dimensional rendering; Output module, used to output the four-dimensional artistic conception after four-dimensional rendering; Using tensor smoothing technology, the material in the three-dimensional artistic conception is driven by action, and the intermediate process of the action is smoothed to generate a four-dimensional time series to achieve four-dimensional rendering, including: Use the size, position and front-to-back relationship of the material in the image to place the three-dimensional material, and use the actions in the action library to drive the movement of the material; after adding the action, a four-dimensional tensor V (X, Y, Z, T) is generated to represent the four-dimensional artistic conception to reflect the change of the three-dimensional artistic conception over time. For multi-channel images, a feature dimension F is added; Three-dimensional artistic conception at any time V t It is expressed as follows: Among them, R1, R2 and R3 represent the number of low-rank components after decomposition of the four dimensions of three-dimensional space plus time; and Represent the low-rank components respectively; Represents a pixel matrix composed in the XY plane; Represents the pixel matrix composed in the XZ plane; Represents the pixel matrix composed in the ZY plane; Indicates V t The vector along the Z axis; Indicates V t The vector along the Y axis; Indicates V t The vector along the X axis; Indicates V t A vector of dimension F along the XY plane; Indicates V t A vector of dimension F along the XZ plane; Indicates V t The vector of dimension F along the YZ plane; f k (t) r Represents the weight of the corresponding low-rank component at time point t, k∈[1,3]; Use the MLP function to Map them separately to obtain the feature D after neural network processing, as follows: in, Represents the pixel matrix composed in the ZT plane; Represents the pixel matrix composed in the YT plane; Represents the pixel matrix composed in the XT plane; The pixels on the plane are stacked to obtain the feature representation D(x, y, z, t) of any pixel point, as follows: in, express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stacking of all pixels in the pixel matrix; express The stack of all pixels in ; express The stack of all pixels in ; express The stack of all pixels in ; Then and Stack to V RF , get the feature representation of any pixel as follows: Among them, the semicolon means connecting all the features; The loss function is minimized for the adjacent pixels on the six surfaces to obtain the smoothed four-dimensional artistic conception pixel points.

9. A computer device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Video generation method and device, electronic equipment and storage medium

    CN114254157A

  • System for intelligently generating ancient poetry situation video through AI and working method thereof

    CN116049523A

  • Method and system for automatically generating crime scene video based on AI voice

    CN117392289A

  • Method, system and equipment for generating movie video clip by text

    CN117478978A

  • Method and processing unit for creating and rendering synchronized content for content rendering environment

    US20230336608A1