Object script determination method, device, system and computer program product

Abstract

The present disclosure relates to a subject script determination method, device, system and computer program product. In at least one embodiment of the present disclosure, by obtaining multi-modal description information of a target object, the multi-modal description information can be fused into a description semantic of the target object, which fuses part of the description information of different modalities and realizes semantic representation of the target object; by obtaining a target background material of the target object, at least one script display area is set on the target background material so as to determine the position of the object script on the target background material; thus, the object script information displayed in each script display area on the target background material is determined by using the description semantic of the target object, without the need for the creator to edit different object scripts at different positions of the background material, thereby reducing the creation threshold and improving the creation efficiency.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, in particular, to a method, device and system for determining object script and computer program product. BACKGROUND

[0002] The existing generation method of object script is that a set of attributes of an object is input into a pre-trained generation model, and the generation model outputs the object script corresponding to the set of attributes.

[0003] However, an object is usually displayed through background material (such as an image, a video, etc.), and the object script needs to be added to the background material in order to highlight the characteristics of the object and quickly capture the attention of users. However, the existing generation method of object script cannot determine the object script in the background material, that is, the generated script is irrelevant to and disconnected from the background material, and still requires the creator to edit different object scripts at different positions of the background material, which has a high requirement for creation, results in a high creation threshold and low creation efficiency. SUMMARY

[0004] At least one embodiment of the present disclosure provides a method, device, system and computer program product for determining object script.

[0005] In a first aspect, embodiments of the present disclosure provide a method for determining object script, which comprises:

[0006] obtaining multi-modal description information of a target object, each modal description information in the multi-modal description information records part of the description information of the target object, and different modal description information is stored in different data storage formats;

[0007] obtaining target background material of the target object, the target background material is provided with at least one script display area;

[0008] based on the multi-modal description information, performing semantic fusion on the part of the description information of the target object recorded by each modal description information to obtain the description semantics of the target object;

[0009] based on the description semantics of the target object, determining the object script information displayed in each script display area on the target background material.

[0010] In some embodiments, obtaining the target background material of the target object comprises:

[0011] obtaining background material of the target object;

[0012] determining the number of script display areas included in the background material and the position of each script display area in the background material;

[0013] Based on the position of each text display area in the background material, the corresponding text display area is set on the background material to obtain a target background material.

[0014] In some embodiments, determining the number of text display areas included in the background material and the position of each text display area in the background material comprises:

[0015] Through the pre-trained text display area determination model, the background material is processed for text display area, and the number of text display areas included in the background material and the position of each text display area in the background material are output.

[0016] Or, in response to the operation of setting the text display area on the background material, the number of text display areas included in the background material and the position of each text display area in the background material are determined.

[0017] In some embodiments, based on the multi-modal description information, the partial description information of the target object recorded by each kind of modal description information is semantically fused to obtain the description semantics of the target object, comprising:

[0018] The multi-modal description information is input into the pre-trained multi-modal semantic understanding model, and the partial description information of the target object recorded by each kind of modal description information is semantically fused by the multi-modal semantic understanding model, and the description semantics of the target object is output.

[0019] In some embodiments, based on the description semantics of the target object, the object text information displayed in each text display area on the target background material is determined, comprising:

[0020] The description semantics of the target object and the target background material are input into the pre-trained text determination model, the text determination is performed by the text determination model, and the object text information displayed in each text display area on the target background material is output.

[0021] In some embodiments, the multi-modal semantic understanding model and the text determination model are trained in a joint training manner, wherein the joint training manner comprises:

[0022] A plurality of training samples are obtained, each training sample corresponds to an object, and each training sample comprises: multi-modal description information, background material provided with at least one text display area, and object text information displayed in each text display area;

[0023] The multi-modal description information of the same object is input into the first model, and the output of the first model and the background material provided with at least one text display area of the same object are input into the second model;

[0024] The object script information displayed in each script display area of the same object is taken as a label, and the parameters of the first model and the parameters of the second model are trained until the output of the second model converges or is equal to the label, and then the training is completed, the first model is trained to obtain a multi-modal semantic understanding model, and the second model is trained to obtain a script determination model.

[0025] In some embodiments, the method further comprises:

[0026] The object script information displayed in each script display area on the target background material is added to the corresponding script display area to generate a target background material with object script information.

[0027] In a second aspect, the embodiments of the present disclosure further provide a script determination device for an object, and the device comprises:

[0028] A first obtaining unit is configured to obtain multi-modal description information of a target object, each modal description information in the multi-modal description information recording part of the description information of the target object, and different modal description information being stored in different data storage formats;

[0029] A second obtaining unit is configured to obtain a target background material of the target object, the target background material being provided with at least one script display area;

[0030] A multi-modal semantic understanding unit is configured to perform semantic fusion on the part of the description information of the target object recorded by each modal description information based on the multi-modal description information to obtain the description semantics of the target object;

[0031] A script determination unit is configured to determine the object script information displayed in each script display area on the target background material based on the description semantics of the target object.

[0032] In a third aspect, the embodiments of the present disclosure further provide a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the steps of the script determination method for an object according to any one of the embodiments of the first aspect.

[0033] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the script determination method for an object according to the first aspect.

[0034] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, which comprises a computer program stored in a computer readable storage medium, and at least one processor of a computer reads and executes the computer program from the storage medium, so that the computer executes the steps of the object script determination method according to any one of the first aspect.

[0035] It can be seen that, in at least one embodiment of the present disclosure, by acquiring the multi-modal description information of the target object, the multi-modal description information can be fused into the description semantics of the target object, the description semantics fuses part of the description information of different modalities, and the semantic representation of the target object is realized; by acquiring the target background material of the target object, at least one script display area is set on the target background material, so as to determine the position of the object script on the target background material; thus, the object script information displayed in each script display area on the target background material is determined by using the description semantics of the target object, and the creator does not need to edit different object scripts at different positions of the background material, the creation threshold is reduced, and the creation efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0037] Figure 1 A flowchart of an object script determination method provided by an embodiment of the present disclosure;

[0038] Figure 2 A flowchart of acquiring a target background material of a target object provided by an embodiment of the present disclosure;

[0039] Figure 3 A flowchart of an object script determination method provided by an embodiment of the present disclosure;

[0040] Figure 4 A schematic diagram of an object script determination device provided by an embodiment of the present disclosure;

[0041] Figure 5 An exemplary block diagram of a system comprising at least one computing device and at least one storage device storing instructions provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In order to enable a more clear understanding of the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It can be understood that the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The specific embodiments described herein are only used to explain the present disclosure, but not to limit the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present disclosure.

[0043] It should be noted that, in this paper, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0044] With the development of network technology, e-commerce (short for e-commerce) has emerged. E-commerce uses the network to carry out transaction activities and related service activities in an electronic transaction manner, so that each link of traditional commercial activities is electrified and networked.

[0045] Merchants can publish new objects (items), and users can browse the object descriptions of the new objects published by the merchants in the store page of the merchant. The published objects usually include two categories: one is physical goods, such as clothing, electrical appliances, and wearable tangible goods, which are usually provided by e-commerce service systems such as Taobao, Pinduoduo, and Jingdong; the other is virtual goods, such as service goods, including food delivery, video, music, and content intangible goods. Whether it is a physical good or a virtual good, it is a labor result produced for sale and a labor product for exchange. Therefore, the published object can be understood as a product for sale.

[0046] The application scenarios of publishing objects usually include two types: one is that the merchant enters the third-party platform, such as the e-commerce service system of Taobao, Jingdong, Amazon, and Pinduoduo; the other is that the merchant builds an e-commerce website. Both of these two application scenarios have the need for generating object scripts. The existing object script generation method is automatic generation, specifically: taking a set of attributes of an object as input to a pre-trained generation model, and outputting the object script corresponding to the set of attributes from the generation model.

[0047] However, the object is usually displayed through background materials (such as images, videos, etc.), and the object text needs to be added to the background materials to highlight the characteristics of the object and quickly capture the attention of the user. However, the existing object text generation method cannot determine the object text in the background materials, and still requires the creator to edit different object texts at different positions of the background materials, which has a high requirement for creation: not only does the creator need to have excellent text foundation, but also the creator needs to be able to grasp the object characteristics that the user cares about and the organization ability of the picture (such as placing different types of texts (selling points, attributes, etc.) at different positions of the background materials to achieve visual beauty and consistency), resulting in a high creation threshold and low creation efficiency.

[0048] Therefore, at least one embodiment of the present disclosure provides a method, device, system, medium or computer program product for determining object text, by obtaining multi-modal description information of a target object, the multi-modal description information can be fused into description semantics of the target object, which fuses part of the description information of different modalities, and realizes semantic representation of the target object; by obtaining target background materials of the target object, at least one text display area is set on the target background materials to determine the position of the object text on the target background materials; thus, the description semantics of the target object is used to determine the object text information displayed in each text display area on the target background materials, without the need for the creator to edit different object texts at different positions of the background materials, thereby reducing the creation threshold and improving the creation efficiency.

[0049] Figure 1 A flowchart of a method for determining object text is provided for the embodiments of the present disclosure, and the execution subject of the method is an electronic device. The electronic device can be a portable mobile device such as a smartphone, a tablet computer, a notebook computer, and smart sports equipment; or a fixed device such as a personal computer, a smart home appliance, and a server, wherein the server can be a single server, a server cluster, a distributed cluster, or a centralized cluster.

[0050] As shown in Figure 1 The method for determining object text can include, but is not limited to, the following steps 101 to 104:

[0051] In step 101, multi-modal description information of a target object is obtained, each type of modal description information in the multi-modal description information records part of the description information of the target object, and different modal description information is stored in different data storage formats.

[0052] The target object is an object to be edited, and the multi-modal description information includes at least one of the following: text, table, non-background picture, non-background video, audio, etc. The text can record the title of the target object, the related information of the target object, and other text information. The table can record the attribute information of the target object itself. The non-background picture is, for example, a picture related to the target object. The non-background video is, for example, a video related to the target object. The audio is, for example, an introduction audio of the target object.

[0053] For example, the target object is a commodity, the text can record the title of the commodity, the related information (brand, applicable age, style, serial number, applicable season, sales channel, etc.) of the commodity, and other text information. The table can record the attribute information (for example, commodity parameter information, including but not limited to: material, color, composition, size, etc.) of the commodity itself. The non-background picture is, for example, a picture containing the commodity. The non-background video is, for example, a video containing the commodity. The audio is, for example, an introduction audio of the commodity.

[0054] In some embodiments, the way to obtain the multi-modal description information of the target object is to obtain the object link of the target object, open the object display page corresponding to the object link, and extract the multi-modal description information of the object from the object display page.

[0055] In other embodiments, the way to obtain the multi-modal description information of the target object is to obtain the object identification (ID) of the target object, and obtain the multi-modal description information corresponding to the object identification from a pre-established object library. The object library stores the correspondence between the object identification and the multi-modal description information.

[0056] In step 102, the target background material of the target object is obtained, and at least one text display area is set on the target background material.

[0057] The target background material can be understood as a background material (such as a picture or a video, etc.) used to carry the object text. That is, the object text is displayed on the target background material.

[0058] The text display area can be represented by a text box. That is, at least one text box is set on the target background material, each text box is used to display different object text information, and different text boxes are located at different positions on the target background material.

[0059] It can be seen that by obtaining the target background material of the target object, at least one text box is set on the target background material, so as to determine the position of the object text on the target background material.

[0060] In step 103, based on the multi-modal description information, the semantic fusion of the part of the description information of the target object recorded by each kind of modal description information is performed to obtain the description semantics of the target object.

[0061] In this embodiment, since each modality description information in the multi-modal description information records part of the description information of the target object, by performing semantic fusion on the part of the description information of the target object recorded in each modality description information, the description semantics of the target object is obtained, which fuses the part of the description information of different modalities, and realizes semantic representation of the target object.

[0062] For example, the target object is a commodity, the multi-modal description information includes text and table, the text records the title of the commodity (for example, female trousers), and the related information of the commodity (for example, the applicable age is 18 to 24 years old, the style is retro, and the applicable season is spring and autumn); the table records the attribute information of the commodity itself (for example, the commodity parameter information, including but not limited to: the material is polyester, the color is black and white, and the trouser type is Harrington trousers, etc.). Correspondingly, the text and the table are semantically fused to obtain the description semantics of the commodity, for example: the commodity is female Harrington trousers, applicable to 18 to 24 years old, retro style, applicable to spring and autumn season, polyester material, and color is black and white.

[0063] It should be noted that the above is only an example for illustration. In actual application, the multi-modal description information is often more complex, but after semantic fusion, the relatively complete description semantics of the commodity can be obtained, wherein the relatively complete is for the multi-modal description information, and if new description information is added, the commodity description semantics obtained after semantic fusion will change accordingly.

[0064] In step 104, based on the description semantics of the target object, the object script information displayed in each script display area on the target background material is determined.

[0065] By using the description semantics of the target object to determine the object script information displayed in each script display area on the target background material, the creator does not need to edit different object scripts at different positions of the background material, thereby reducing the creation threshold and improving the creation efficiency.

[0066] For example, the object is a commodity, and by obtaining the multi-modal description information of the commodity, the multi-modal description information can be fused into the description semantics of the commodity, which fuses the part of the description information of different modalities, realizes semantic representation of the commodity, and understands the selling points of the commodity. By obtaining the target background material of the commodity, at least one text box is set on the target background material to determine the position of the commodity script on the target background material. Thus, by using the description semantics of the commodity to determine the commodity script information displayed in each text box on the target background material, the creator does not need to edit different commodity scripts at different positions of the background material, thereby reducing the creation threshold and improving the creation efficiency.

[0067] Based on Figure 1The object script determination method shown in some embodiments, Figure 2 A flowchart for obtaining a target background material of a target object is provided for the embodiments of the present disclosure, as shown in Figure 2 As shown, obtaining a target background material of a target object includes the following steps 201 to 203:

[0068] In step 201, the background material of the target object is obtained.

[0069] In this embodiment, the background material of the target object can be specified by the creator, and the background material includes the target object. For example, the creator specifies one material as the background material from a plurality of materials (such as a plurality of pictures or videos including the target object) including the target object.

[0070] In step 202, the number of script display areas included in the background material and the position of each script display area in the background material are determined.

[0071] In this embodiment, the pre-trained script display area determination model is used to process the script display area of the background material, and the number of script display areas included in the background material and the position of each script display area in the background material are output.

[0072] The pre-trained script display area determination model is a machine learning model, such as a neural network model, and the training process is as follows:

[0073] The training sample is obtained, which is a background material with known number of text boxes and position of each text box; then the training sample is input into the neural network model, and the parameters of the neural network model itself are continuously adjusted until the output of the neural network model is the same as or converges to the known number of text boxes and positions of the training sample, and the training is completed, obtaining the script display area determination model.

[0074] In some other embodiments, in response to an operation of setting a script display area on the background material, the number of script display areas included in the background material and the position of each script display area in the background material are determined. The operation of setting a script display area on the background material is a manual operation, for example, the number and position of the script display area on the background material are set by the creator.

[0075] In step 203, based on the position of each script display area in the background material, the corresponding script display area is set on the background material to obtain the target background material.

[0076] The target background material can be understood as a background material with a script display area.

[0077] Based on Figure 1The object script determination method shown in some embodiments, Figure 1 In step 103, the part of the description information of the target object recorded by each modality description information is semantically fused based on the multi-modal description information to obtain the description semantics of the target object, specifically:

[0078] The multi-modal description information is input into a pre-trained multi-modal semantic understanding model, and the part of the description information of the target object recorded by each modality description information is semantically fused by the multi-modal semantic understanding model to output the description semantics of the target object.

[0079] The input of the multi-modal semantic understanding model is the multi-modal description information of the target object, and the output of the multi-modal semantic understanding model is the description semantics of the target object. In some embodiments, the multi-modal semantic understanding model can be any type of model capable of receiving multi-modal data input, such as a multi-modal multi-copy grid (M4C) model.

[0080] Based on Figure 1 The object script determination method shown in some embodiments, Figure 1 In step 104, the object script information displayed in each script display area on the target background material is determined based on the description semantics of the target object, specifically:

[0081] The description semantics of the target object and the target background material are input into a pre-trained script determination model, and the script determination model is used for script determination to output the object script information displayed in each script display area on the target background material.

[0082] The input of the script determination model is the description semantics of the target object (i.e., the input of the multi-modal semantic understanding model) and the target background material, and the output of the script determination model is the object script information displayed in each script display area on the target background material. In some embodiments, the script determination model can be any type of language model, such as a generative pre-training (GPT) model.

[0083] On the basis of the above embodiments, the multi-modal semantic understanding model and the script determination model are trained in a joint training manner, wherein the joint training manner includes the following steps A to C:

[0084] Step A, obtaining a plurality of training samples, each training sample corresponding to an object, and each training sample including: multi-modal description information, background material provided with at least one script display area, and object script information displayed in each script display area; wherein the number and position of the script display areas on the background material are designed by professional personnel according to the aesthetic degree and the degree of attracting attention of the background material.

[0085] Step B: Input the multimodal description information of the same object into the first model, and input the output of the first model and the background material of the same object with at least one text display area into the second model;

[0086] Step C: Using the text information of the object displayed in each text display area of ​​the same object as a label, train the parameters of the first model and the second model until the output of the second model converges to or equals the label. Then the training is completed. The first model is trained to obtain a multimodal semantic understanding model, and the second model is trained to obtain a text determination model.

[0087] The first model can be any type of model capable of receiving multimodal data input, such as a multimodal multicopy grid (M4C) model. The second model can be any type of language model, such as a generative pre-training (GPT) model.

[0088] based on Figure 1 The method for determining the object text shown may, in some embodiments, include the following steps:

[0089] Add the object text information displayed in each text display area of ​​the target background material to the corresponding text display area to generate a target background material with object text information.

[0090] Based on the above embodiments, creators can be provided with the ability to add text to background materials with one click. For example, a one-click add control can be provided, and in response to the creator's operation of triggering the one-click add control, the following can be executed: Figure 1 The method for determining the object text shown obtains the object text information displayed in each text display area on the target background material; and adds the object text information displayed in each text display area on the target background material to the corresponding text display area to generate a target background material with object text information. It is evident that this embodiment can automatically add text to background materials, lowering the creative threshold and improving creative efficiency.

[0091] Figure 3 This is a flowchart illustrating a method for determining object text provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, the process for determining the target text is as follows:

[0092] The multi-modal description information of the target object is obtained, the multi-modal description information is input into a multi-modal semantic understanding model, the multi-modal semantic understanding model performs semantic fusion on part of the description information of the target object recorded by each kind of modal description information to obtain description semantics of the target object, the multi-modal semantic understanding model inputs the description semantics of the target object into a script determination model, and inputs target background material of the target object into the script determination model, wherein the target background material of the target object is a picture, and the picture includes two text boxes, the script determination model performs script determination to obtain object script information displayed in each text box on the target background material, and the object script information displayed in each text box on the target background material is added into the corresponding text box to generate the target background material with the object script information.

[0093] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art can understand that the embodiments of the present disclosure are not limited by the action sequence described, because according to the embodiments of the present disclosure, certain steps can be performed in other sequences or simultaneously. In addition, those skilled in the art can understand that the embodiments described in the specification all belong to optional embodiments.

[0094] Figure 4 A schematic diagram of an object script determination device provided by the embodiments of the present disclosure is shown in FIG. 1. The execution subject of the object script determination device is an electronic device. The electronic device can be a portable mobile device such as a smart phone, a tablet computer, a notebook computer, and smart sports equipment; or a fixed device such as a personal computer, a smart home appliance, and a server. The server can be a single server, a server cluster, a distributed cluster, or a centralized cluster. Figure 4 As shown in FIG. 1, the object script determination device can include but is not limited to a first obtaining unit 41, a second obtaining unit 42, a multi-modal semantic understanding unit 43, and a script determination unit 44.

[0095] The first obtaining unit 41 is configured to obtain multi-modal description information of a target object. Each kind of modal description information in the multi-modal description information records part of the description information of the target object, and different modal description information is stored in different data storage formats.

[0096] The second obtaining unit 42 is configured to obtain target background material of the target object. The target background material is provided with at least one script display area.

[0097] The multi-modal semantic understanding unit 43 is configured to perform semantic fusion on part of the description information of the target object recorded by each kind of modal description information based on the multi-modal description information to obtain description semantics of the target object.

[0098] The script determination unit 44 is configured to determine object script information displayed in each script display area on the target background material based on the description semantics of the target object.

[0099] In some embodiments, the second acquisition unit 42 is configured to: acquire the background material of the target object; determine the number of script display areas included in the background material and the position of each script display area in the background material; and set a corresponding script display area on the background material based on the position of each script display area in the background material, to obtain the target background material.

[0100] In some embodiments, the second acquisition unit 42 is configured to determine the number of script display areas included in the background material and the position of each script display area in the background material, by:

[0101] The script display area determination model is pre-trained, and the script display area determination model is used to process the script display area of the background material, to output the number of script display areas included in the background material and the position of each script display area in the background material.

[0102] Alternatively, the number of script display areas included in the background material and the position of each script display area in the background material are determined in response to an operation of setting a script display area on the background material.

[0103] In some embodiments, the multi-modal semantic understanding unit 43 is configured to input the multi-modal description information into a pre-trained multi-modal semantic understanding model, and perform semantic fusion on part of the description information of the target object recorded by each kind of modal description information by using the multi-modal semantic understanding model, to output the description semantics of the target object.

[0104] In some embodiments, the script determination unit 44 is configured to input the description semantics of the target object and the target background material into a pre-trained script determination model, and perform script determination by using the script determination model, to output object script information displayed in each script display area on the target background material.

[0105] In some embodiments, the multi-modal semantic understanding model and the script determination model are trained in a joint training manner, and the joint training manner includes:

[0106] A plurality of training samples are acquired, each training sample corresponds to an object, and each training sample includes: multi-modal description information, a background material provided with at least one script display area, and object script information displayed in each script display area.

[0107] The multi-modal description information of the same object is input into the first model, and the output of the first model and the background material provided with at least one script display area of the same object are input into the second model.

[0108] The object script information displayed in each script display area of the same object is taken as a label, and the parameters of the first model and the parameters of the second model are trained until the output of the second model converges or is equal to the label, and then the training is completed, the first model is trained to obtain a multi-modal semantic understanding model, and the second model is trained to obtain a script determination model.

[0109] In some embodiments, the object script determination apparatus further comprises a generation unit configured to add the object script information displayed in each script display area of the target background material into the corresponding script display area, and generate the target background material with the object script information.

[0110] Details of each of the above disclosed embodiments of the object script determination apparatus can refer to details of each of the above disclosed embodiments of the object script determination method, and will not be repeated here.

[0111] Figure 5 is an exemplary block diagram of a system including at least one computing device and at least one storage device storing instructions, provided by an embodiment of the present disclosure. In some embodiments, the system can be used for big data processing, and the at least one computing device and the at least one storage device can be distributedly deployed, so that the system is a distributed data processing cluster.

[0112] As shown in Figure 5 , the system includes at least one computing device 51 and at least one storage device 52 storing instructions. It can be understood that the storage device 52 in the embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0113] In some embodiments, the storage device 52 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and an application program.

[0114] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic tasks and processing hardware-based tasks. The application program includes various application programs, such as a media player (Media Player), a browser (Browser), etc., for implementing various application tasks. The program for implementing the object script determination method provided by the embodiment of the present disclosure can be included in the application program.

[0115] In the embodiment of the present disclosure, the at least one computing device 51 invokes the program or instruction stored in the at least one storage device 52, specifically, the program or instruction stored in the application program, and the at least one computing device 51 is configured to execute the steps of each of the embodiments of the object script determination method provided by the embodiment of the present disclosure.

[0116] The method for determining an object script provided in the embodiments of the present disclosure can be applied to or implemented by the computing device 51. The computing device 51 can be an integrated circuit chip with a signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the computing device 51. The computing device 51 described above can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0117] The steps of the method for determining an object script provided in the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution or a combination of hardware and software in the decoding processor for execution. The software unit can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the field. The storage medium is located in the storage device 52, and the computing device 51 reads the information in the storage device 52 and combines the hardware thereof to complete the steps of the method.

[0118] The embodiments of the present disclosure further provide a computer readable storage medium storing a program or instruction, which, when executed by at least one computing device, causes the at least one computing device to perform the steps of the method for determining an object script. For the sake of avoiding repetitive description, the steps will not be described herein. The computing device can be the computing device 51 shown in the figure. In some embodiments, the computer readable storage medium is a non-transitory computer readable storage medium. Figure 5

[0119] The embodiments of the present disclosure further provide a computer program product, which includes a computer program stored in a non-transitory computer readable storage medium. At least one processor of a computer reads and executes the computer program from the storage medium, so that the computer performs the steps of the method for determining an object script. For the sake of avoiding repetitive description, the steps will not be described herein.

[0120] ​It has to be noted that, as used herein, the terms "includes", "including", "to include", "includes" or "including" and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without further restriction, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0121] Those skilled in the art will appreciate that the features of the various embodiments can be combined with each other, where appropriate, in accordance with the disclosure and form different embodiments.

[0122] Those skilled in the art will appreciate that the description of various embodiments have been with emphasis on the principles of the disclosure. Parts of the description that have not been fully detailed are readily apparent to those skilled in the art in view of the disclosure.

[0123] Although the embodiments of the disclosure have been described with reference to the accompanying drawings, various modifications and changes can be suggested to one skilled in the art, and it is intended that the disclosure encompass such modifications and changes as fall within the scope of the appended claims.

Claims

1. A method for determining an object script, the method comprising: obtaining multi-modal description information of a target object, each of the multi-modal description information recording partial description information of the target object, and different modal description information being stored in different data storage formats; obtaining a target background material of the target object, at least one script display area being set on the target background material based on a number of script display areas and a position of each script display area; based on the multi-modal description information, performing semantic fusion on the partial description information of the target object recorded by each modal description information to obtain description semantics of the target object; inputting the description semantics of the target object and the target background material into a pre-trained script determination model, performing script determination by the script determination model, and outputting object script information displayed in each script display area on the target background material, the script determination model being obtained based on a plurality of training samples, each of the training samples corresponding to an object, and each of the training samples comprising a background material provided with at least one script display area and object script information displayed in each script display area.

2. The method of claim 1, wherein, The obtaining of the target background material of the target object comprises: obtaining a background material of the target object; determining a number of script display areas included in the background material and a position of each script display area in the background material; based on the position of each script display area in the background material, setting a corresponding script display area on the background material to obtain a target background material.

3. The method of claim 2, wherein, The determination of the number of script display areas included in the background material and the position of each script display area in the background material comprises: performing script display area processing on the background material by a pre-trained script display area determination model to output the number of script display areas included in the background material and the position of each script display area in the background material; or in response to an operation of setting a script display area on the background material, determining the number of script display areas included in the background material and the position of each script display area in the background material.

4. The method of claim 1, wherein, The semantic fusion on the partial description information of the target object recorded by each modal description information based on the multi-modal description information to obtain the description semantics of the target object comprises: inputting the multi-modal description information into a pre-trained multi-modal semantic understanding model, performing semantic fusion on the partial description information of the target object recorded by each modal description information by the multi-modal semantic understanding model, and outputting the description semantics of the target object.

5. The method of claim 4, wherein, The multi-modal semantic understanding model and the script determination model are trained in a joint training manner, wherein the joint training manner comprises: obtaining a plurality of training samples, each of the training samples corresponding to an object, and each of the training samples comprising multi-modal description information, a background material provided with at least one script display area, and object script information displayed in each script display area. inputting the multi-modal description information of the same object into a first model, and inputting an output of the first model and a background material of the same object provided with at least one script display region into a second model; training parameters of the first model and parameters of the second model until the output of the second model converges or equals to the label, wherein the first model is trained to obtain the multi-modal semantic understanding model, and the second model is trained to obtain the script determination model.

6. The method of claim 1, wherein, The method further comprises: adding the object script information displayed in each script display region of the target background material into the corresponding script display region to generate the target background material with the object script information.

7. A script determination apparatus for an object, the apparatus comprising: a first obtaining unit configured to obtain multi-modal description information of a target object, each modal description information of the multi-modal description information recording partial description information of the target object, and different modal description information being stored in different data storage formats; a second obtaining unit configured to obtain a target background material of the target object, the target background material being provided with at least one script display region based on a number of script display regions and a position of each script display region; a multi-modal semantic understanding unit configured to perform semantic fusion on the partial description information of the target object recorded by each modal description information based on the multi-modal description information to obtain description semantics of the target object; a script determination unit configured to input the description semantics of the target object and the target background material into a pre-trained script determination model, perform script determination through the script determination model, and output object script information displayed in each script display region of the target background material, wherein the script determination model is obtained based on a plurality of training samples, each training sample corresponding to an object, and each training sample comprising a background material provided with at least one script display region and object script information displayed in each script display region.

8. A system comprising at least one computing device and at least one storage device storing instructions, wherein, The instructions, when executed by the at least one computing device, cause the at least one computing device to perform the steps of the method of determining a script for an object according to any one of claims 1 to 6.

9. A computer readable storage medium, wherein, The computer-readable storage medium stores programming or instructions that, when executed by at least one computing device, cause the at least one computing device to perform the steps of the method of determining a script for an object according to any one of claims 1 to 6.

10. A computer program product, wherein, The computer program product comprises a computer program stored in a computer-readable storage medium, at least one processor of a computer reading and executing the computer program from the storage medium, so that the computer performs the steps of the method of determining a script for an object according to any one of claims 1 to 6.

Images