Data processing method and device

Abstract

The invention discloses a data processing method and device, and the method comprises the steps: obtaining input data for a three-dimensional scene, and enabling the input data to comprise information corresponding to a target object; target position information is obtained according to scene semantic information of the three-dimensional scene, the input data and a target model, the target position information represents the position of the target object in the three-dimensional scene, and the target model can generate an output result according to a received natural language instruction; determining target view angle information according to the target position information; and rendering a target image corresponding to the target view angle information through a three-dimensional model corresponding to the three-dimensional scene.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method and device. Background Art

[0002] Users can use artificial intelligence models to process 3D scenes, for example, adding objects to the 3D scene or modifying the color or placement of objects in the 3D scene.

[0003] However, this requires the user to manually operate the three-dimensional scene first, such as dragging the perspective of the three-dimensional scene, and then presenting the object from the three-dimensional scene before proceeding to the next operation, which results in high complexity of user operation. Summary of the invention

[0004] In view of this, the present application provides a data processing method and device as follows:

[0005] A data processing method, comprising:

[0006] Obtaining input data for a three-dimensional scene, wherein the input data includes information corresponding to a target object;

[0007] Obtaining target position information according to scene semantic information of the three-dimensional scene, the input data and the target model, wherein the target position information represents the position of the target object in the three-dimensional scene, and the target model can generate an output result according to the received natural language instruction;

[0008] Determining target viewing angle information according to the target position information;

[0009] A target image corresponding to the target viewing angle information is rendered using the three-dimensional model corresponding to the three-dimensional scene.

[0010] The above method preferably obtains the target position information according to the scene semantic information of the three-dimensional scene, the input data and the target model, including:

[0011] Inputting the input data into a target model to obtain target position information output by the target model;

[0012] Wherein, the target model is optimized using scene semantic information of the three-dimensional scene.

[0013] The above method preferably obtains the target position information according to the scene semantic information of the three-dimensional scene, the input data and the target model, including:

[0014] Obtaining first prompt information according to the scene semantic information of the three-dimensional scene and the input data;

[0015] The first prompt information is input into a target model to obtain target position information output by the target model.

[0016] Preferably, in the above method, after rendering the target image corresponding to the target viewing angle information by using the three-dimensional model corresponding to the three-dimensional scene, the method further comprises:

[0017] According to the operation prompt information, perform corresponding operations on the target object;

[0018] The operation prompt information is obtained at least according to the operation text of the three-dimensional model, the input data and the target model, and the operation text represents the operation process corresponding to the three-dimensional model.

[0019] In the above method, preferably, the operation prompt information is obtained by:

[0020] Inputting the input data into the target model to obtain operation prompt information output by the target model;

[0021] Wherein, the target model is optimized using the operation text and the scene semantic information.

[0022] In the above method, preferably, the operation prompt information is obtained by:

[0023] Extracting key operation information from the operation text;

[0024] Obtaining second prompt information according to the key operation information, the scene semantic information and the input data;

[0025] The second prompt information is input into the target model to obtain operation prompt information output by the target model.

[0026] The above method, preferably, further comprises:

[0027] The scene semantic information is updated according to the input data and the operation prompt information.

[0028] In the above method, preferably, the scene semantic information is obtained by:

[0029] Obtaining scanned images of the three-dimensional scene at multiple viewing angles;

[0030] Performing semantic analysis on the visual content in the scanned image to obtain scene semantic information;

[0031] The scene semantic information includes at least one of the following:

[0032] Scene description information of the three-dimensional scene;

[0033] Object description information of objects included in the three-dimensional scene;

[0034] Three-dimensional coordinate information of objects contained in the three-dimensional scene;

[0035] The relative position relationship between the objects included in the three-dimensional scene.

[0036] The above method, preferably, further comprises:

[0037] If the current viewing angle information of the three-dimensional scene is different from the viewing angle information corresponding to the scanned image, performing image scanning on the three-dimensional scene under the current viewing angle information to obtain a new scanned image;

[0038] The scene semantic information is updated according to the new scanned image.

[0039] A data processing device, comprising:

[0040] A data acquisition unit, used to obtain input data for a three-dimensional scene, wherein the input data includes information corresponding to a target object;

[0041] a position obtaining unit, configured to obtain target position information according to scene semantic information of the three-dimensional scene, the input data and a target model, wherein the target position information represents the position of the target object in the three-dimensional scene, and the target model can generate an output result according to a received natural language instruction;

[0042] A viewing angle obtaining unit, used to determine target viewing angle information according to the target position information;

[0043] An image rendering unit is used to render a target image corresponding to the target viewing angle information using a three-dimensional model corresponding to the three-dimensional scene.

[0044] A computer program product includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the data processing methods described above.

[0045] An electronic device comprises at least one processor and a memory connected to the processor, wherein:

[0046] The memory is used to store computer programs;

[0047] The processor is used to execute the computer program so that the electronic device can implement any one of the data processing methods described above.

[0048] A computer storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the data processing methods described above.

[0049] It can be seen from the above technical solutions that in a data processing method and device disclosed in the present application, after obtaining input data for a three-dimensional scene, the target position information representing the position of the target object in the three-dimensional scene can be obtained according to the scene semantic information of the three-dimensional scene, the input data, and the target model that can generate output results according to the received natural language instructions. In this way, the target perspective information is determined according to the target position information, and the target image corresponding to the target perspective information is rendered through the three-dimensional model corresponding to the three-dimensional scene. It can be seen that in the present application, the target image at the position of the target object can be rendered according to the scene semantic information, input data and target model of the three-dimensional scene. For the user, only the information corresponding to the target object needs to be input, and the corresponding target image can be rendered without the user dragging the three-dimensional scene to find the target object, thereby reducing the complexity of the user's operation on the three-dimensional model and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0052] Figure 2 This is an example diagram of rendering a target image according to the perspective of the "display" in the focus position in this application;

[0053] Figure 3 Another flow chart of a data processing method provided in an embodiment of the present application;

[0054] Figure 4 This is an example diagram of operating a target object in a three-dimensional scene in this application;

[0055] Figure 5 A partial flow chart of a data processing method provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0057] Figure 7Another structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0058] Figure 8 Another structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0060] Fig.10 This is a flowchart for implementing a scenario in which the metaverse construction platform performs three-dimensional scene operations applicable to this application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] refer to Figure 1 As shown, it is a flowchart of a data processing method provided in an embodiment of the present application, which can be applied to electronic devices capable of data processing and image rendering, such as mobile phones, tablet devices, notebooks or servers, etc. The technical solution in this embodiment is mainly used to reduce the complexity of the user's operation of the three-dimensional scene and improve the user experience.

[0063] Specifically, the method in this embodiment may include the following steps:

[0064] Step 101: Obtain input data for a three-dimensional scene.

[0065] The input data includes information corresponding to the target object, which may be identification information such as the name or code of the target object that can uniquely represent the target object.

[0066] Specifically, in this embodiment, an interactive interface can be provided to the user through an interactive engine such as a voice assistant, so that the user can input any one or more types of data such as voice, text or image. Based on this, the input data of the user on the interactive interface can be obtained in this embodiment.

[0067] For example, the target object is a display in a three-dimensional scene, and the user can input "find the big screen inside" in the interactive interface. Based on this, the user's input data can be obtained in this embodiment. The input data can be the original content entered by the user in the interactive interface, such as "find the big screen inside"; or, the input data can also be the content after the interactive engine adjusts the original content, such as "find the big screen".

[0068] Step 102: Obtain target position information according to scene semantic information of the three-dimensional scene, input data and a target model.

[0069] The target position information represents the position of the target object in the three-dimensional scene, such as the three-dimensional coordinates of the target object in the three-dimensional scene.

[0070] The target model can generate output results according to the received natural language instructions. For example, the target model can be a large language model (LLM) or a multimodal large model. When the natural language instructions are input to the target model, the target model can generate corresponding output results.

[0071] It should be noted that the scene semantic information of the three-dimensional scene may include at least one of the following:

[0072] Scene description information of the 3D scene, such as "the 3D scene is an indoor office scene, which contains many office supplies and flowers";

[0073] Object description information of objects contained in the 3D scene, such as the shape, posture, and other description information of objects such as “display screen, office chair, bookshelf, keyboard, workstation partition” in the 3D scene;

[0074] 3D coordinate information of objects contained in the 3D scene, such as the 3D coordinate information of the “display screen” and the 3D coordinate information of the “office chair”;

[0075] The relative position relationship between the objects contained in the three-dimensional scene, such as the "monitor" and "bookshelf" are adjacent to each other on the left and right, the "office chair" and "workstation partition" are 1 meter apart, and so on.

[0076] Specifically, in this embodiment, the target model can be used to process scene semantic information and input data of the three-dimensional scene to obtain target position information.

[0077] Among them, the target position information can be the three-dimensional coordinate information of the target object in the three-dimensional scene, or it can also be information such as the relative position relationship between the target object and other objects in the three-dimensional scene. All of this information can represent the position of the target object in the three-dimensional scene.

[0078] In one implementation, in this embodiment, the target model can be optimized in advance using the scene semantic information of the three-dimensional scene. Based on this, in step 102, the input data can be directly input into the target model to obtain the target position information output by the target model.

[0079] For example, taking the target model as a large language model, in this embodiment, the scene semantic information of the three-dimensional scene is used to fine-tune the large language model so that the large language model has the ability to understand the 3D space of the three-dimensional scene. For example, the large language model can understand the shape, posture, three-dimensional coordinates, relative position and other information of each object contained in the three-dimensional scene. Based on this, when the input data "find the big screen inside" is input into the large language model, the large language model can output various information related to the "big screen", such as the shape, posture, three-dimensional coordinates of the "big screen" and the relative position with other objects.

[0080] In another implementation, in step 102, first prompt information may be obtained based on scene semantic information of the three-dimensional scene and input data, and then the first prompt information may be input into the target model to obtain target position information output by the target model.

[0081] Specifically, in this embodiment, the input data can be spliced ​​with the scene semantic information of the three-dimensional scene, so that the scene semantic information of the three-dimensional scene is input into the large language model as the context of the input data. The large language model can parse the intention of the input data based on the scene semantic information of the three-dimensional scene, thereby obtaining the target position information output by the large language model.

[0082] For example, taking the large language model as an example, in this embodiment, the scene semantic information of the three-dimensional scene is spliced ​​as the context of the input data "find the big screen inside" to obtain a first prompt information, which contains at least one prompt word (prompt). These prompt words are then input into the large language model to obtain the large language model output of various information related to the "big screen", such as the shape, posture, three-dimensional coordinates of the "big screen", and relative position with other objects.

[0083] Step 103: Determine target viewing angle information according to the target position information.

[0084] Specifically, in this embodiment, based on the three-dimensional scene and according to the target position information, the viewing angle at which the target object can be seen, that is, the target viewing angle information, can be determined.

[0085] For example, after determining the three-dimensional coordinates of the "display", based on the three-dimensional scene, find the viewing angle that can make the "display" in the focus position, such as Figure 2 As shown in , that is, the target perspective information.

[0086] Step 104: Rendering a target object corresponding to the target viewing angle information using the three-dimensional model corresponding to the three-dimensional scene.

[0087] The 3D model is a tool model that can operate the 3D scene, and the 3D model can render a 3D image of the 3D scene at any viewing angle. Based on this, in step 104, the 3D model is controlled to perform image rendering according to the target viewing angle information, so as to render a 3D image that can present the target object such as the "display" in the middle area, i.e., the target image. Figure 2 as shown in .

[0088] It can be seen from the above technical solution that in a data processing method provided by an embodiment of the present application, after obtaining input data for a three-dimensional scene, the target position information representing the position of the target object in the three-dimensional scene can be obtained according to the scene semantic information of the three-dimensional scene, the input data, and the target model that can generate output results according to the received natural language instructions, so as to determine the target perspective information according to the target position information, thereby rendering the target image corresponding to the target perspective information through the three-dimensional model corresponding to the three-dimensional scene. It can be seen that in this embodiment, the target image at the position of the target object can be rendered according to the scene semantic information, input data and target model of the three-dimensional scene. For the user, only the information corresponding to the target object needs to be input, and the corresponding target image can be rendered without the user dragging the three-dimensional scene to find the target object, thereby reducing the complexity of the user's operation on the three-dimensional model and improving the user experience.

[0089] based on Figure 1 In the implementation scheme shown in FIG. 1 , after step 104, the method in this embodiment may further include the following processing: Figure 3 As shown in:

[0090] Step 105: Perform corresponding operations on the target object according to the operation prompt information.

[0091] The operation prompt information is obtained at least based on the operation text, input data and target model of the three-dimensional model. The operation text represents the operation process corresponding to the three-dimensional model. For example, the operation text can be obtained by extracting key information from the usage method and function description of the three-dimensional model.

[0092] It should be noted that the input data used to obtain the operation prompt information may be the input data obtained in step 101, or may be input data re-obtained relative to the input data in step 101, or may include both the input data obtained in step 101 and the re-obtained input data.

[0093] For example, the input data obtained in step 101 is "a large screen inside displays wedding photos", based on which, the operation prompt information in step 105 is obtained according to the operation text of the three-dimensional model, "a large screen inside displays wedding photos" and the target model;

[0094] For another example, the input data obtained in step 101 is “find the big screen inside”. After the target image is rendered in step 104, new input data “show wedding photos” is obtained in this embodiment. Based on this, the operation prompt information in step 105 is obtained according to the operation text of the three-dimensional model, “show wedding photos” and the target model.

[0095] Specifically, in this embodiment, the target model can be used to process the operation text and input data of the three-dimensional model to obtain operation prompt information.

[0096] In one implementation, the target model can be optimized using the operation text and scene semantic information in this embodiment. Based on this, in this embodiment, the input data can be directly input into the target model to obtain the operation prompt information output by the target model.

[0097] For example, taking the target model as a large language model, in this embodiment, the scene semantic information of the three-dimensional scene and the operation text of the three-dimensional model are used to fine-tune the large language model so that the large language model has the ability to understand the 3D space of the three-dimensional scene and the ability to operate the three-dimensional model. For example, the large language model can understand the shape, posture, three-dimensional coordinates, relative position and other information of each object contained in the three-dimensional scene. The large model can also understand the operation method of implementing any function of the three-dimensional scene in the three-dimensional model. Based on this, when the input data "display wedding photos" is input into the large language model, the large language model outputs operation prompt information. Based on this, in this embodiment, the corresponding operation can be performed on the target object in the three-dimensional scene through the three-dimensional model according to the operation prompt information. For example, the image of "wedding dress" is output on the "display screen" in the three-dimensional scene. Figure 4 as shown in .

[0098] In another implementation method, in this embodiment, key operation information can be first extracted from the operation text, such as keywords for using the three-dimensional model to implement various functions of the three-dimensional scene, and then second prompt information can be obtained based on the key operation information, scene semantic information and input data. Finally, the second prompt information is input into the target model to obtain the operation prompt information output by the target model.

[0099] Specifically, in this embodiment, the input data, scene semantic information, and key operation information extracted from the operation text can be spliced ​​together. In this way, the scene semantic information of the three-dimensional scene and the key operation information extracted from the operation text are input into the large language model together as the context of the input data. The large language model can parse the intention of the input data based on the scene semantic information of the three-dimensional scene and the key operation information extracted from the operation text, thereby obtaining the operation prompt information output by the large language model.

[0100] For example, taking the large language model as an example, in this embodiment, the scene semantic information of the three-dimensional scene and the key operation information extracted from the operation text are spliced ​​as the context of the input data "show wedding photos" and "find the big screen inside" to obtain the second prompt information, and the second prompt information contains multiple prompt words (prompt), and then these prompt words are input into the large language model to obtain the operation prompt information output by the large language model. Based on this, in this embodiment, according to the operation prompt information, the corresponding operation can be performed on the target object in the three-dimensional scene through the three-dimensional model. For example, the image of "wedding dress" is output in the "display screen" in the three-dimensional scene, such as Figure 4 as shown in .

[0101] Based on the above implementation scheme, the present embodiment can also update the scene semantic information according to the input data and operation prompt information. For example, the present embodiment updates the scene semantic information of the three-dimensional scene by accumulating the user's dialogue in the interaction engine and the interaction with the three-dimensional model, and the updated scene semantic information is used in subsequent interactions, thereby providing users with more accurate intelligent services.

[0102] For example, after optimizing the target model using the updated scene semantic information, the target model can understand the three-dimensional scene more accurately, determine the target perspective information more accurately, render the target image more accurately, and accordingly, perform more accurate operations on the target object.

[0103] In one implementation, the scene semantic information of the three-dimensional scene can be realized in the following manner, such as Figure 5 As shown in:

[0104] Step 501: Obtain scanned images of a three-dimensional scene at multiple viewing angles.

[0105] The scanned image may be a two-dimensional image of a three-dimensional scene at a corresponding viewing angle.

[0106] Step 502: semantically analyze the visual content in the scanned image to obtain scene semantic information.

[0107] Specifically, in this embodiment, the scanned image can be input into the visual semantic big model to obtain the scene semantic information output by the visual semantic big model for the scanned image, such as scene description information, object description information of objects such as objects or people, three-dimensional coordinate information of objects such as objects, relative position relationships between objects such as objects, and other information.

[0108] Furthermore, with the use of the three-dimensional scene in the three-dimensional model, the three-dimensional scene may be at any viewing angle. Based on this, the following processing may also be performed in this embodiment:

[0109] If the current viewing angle information of the three-dimensional scene is different from the viewing angle information corresponding to the scanned image, then in this embodiment, the three-dimensional scene can be scanned under the current viewing angle information to obtain a new scanned image, and then the scene semantic information is updated based on the new scanned image.

[0110] For example, a new scanned image is input into the visual semantic big model to obtain the scene semantic information output by the visual semantic big model for the new scanned image, so that the scene semantic information can more accurately represent the three-dimensional scene, and then the target model can understand the three-dimensional scene more accurately based on the scene semantic information, the determined target perspective information is more accurate, the rendered target image is more accurate, and accordingly, more accurate operations can be performed on the target object.

[0111] refer to Figure 6 , is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application, which can be configured in an electronic device capable of data processing and image rendering, such as a mobile phone, a tablet device, a notebook or a server. The device can be a program module capable of calling a target model, such as an agent, which can transmit data to a large language model through an interface and receive the output results of the large language model. The technical solution in this embodiment is mainly used to reduce the complexity of the user's operation on the three-dimensional scene and improve the user experience.

[0112] Specifically, the device in this embodiment may include the following units:

[0113] A data acquisition unit 601 is used to acquire input data for a three-dimensional scene, wherein the input data includes information corresponding to a target object;

[0114] A position obtaining unit 602 is used to obtain target position information according to scene semantic information of the three-dimensional scene, the input data and a target model, wherein the target position information represents the position of the target object in the three-dimensional scene, and the target model can generate an output result according to the received natural language instruction;

[0115] A viewing angle obtaining unit 603 is used to determine target viewing angle information according to the target position information;

[0116] The image rendering unit 604 is configured to render a target image corresponding to the target viewing angle information using a three-dimensional model corresponding to the three-dimensional scene.

[0117] It can be seen from the above technical solution that in a data processing device provided by an embodiment of the present application, after obtaining input data for a three-dimensional scene, the target position information representing the position of the target object in the three-dimensional scene can be obtained according to the scene semantic information of the three-dimensional scene, the input data, and the target model that can generate output results according to the received natural language instructions, so as to determine the target perspective information according to the target position information, thereby rendering the target image corresponding to the target perspective information through the three-dimensional model corresponding to the three-dimensional scene. It can be seen that in this embodiment, the target image at the position of the target object can be rendered according to the scene semantic information, input data and target model of the three-dimensional scene. For the user, only the information corresponding to the target object needs to be input, and the corresponding target image can be rendered without the user dragging the three-dimensional scene to find the target object, thereby reducing the complexity of the user's operation on the three-dimensional model and improving the user experience.

[0118] In one implementation, the position obtaining unit 602 is specifically used to: input the input data into a target model to obtain target position information output by the target model; wherein the target model is optimized using scene semantic information of the three-dimensional scene.

[0119] In one implementation, the position obtaining unit 602 is specifically used to: obtain first prompt information according to the scene semantic information of the three-dimensional scene and the input data; and input the first prompt information into the target model to obtain the target position information output by the target model.

[0120] In one implementation, the device in this embodiment further includes the following units: Figure 7 As shown in:

[0121] The operation execution unit 605 is used to perform a corresponding operation on the target object according to the operation prompt information after the image rendering unit 604 renders the target image corresponding to the target perspective information through the three-dimensional model corresponding to the three-dimensional scene; wherein the operation prompt information is obtained at least according to the operation text, input data and the target model of the three-dimensional model, and the operation text represents the operation process corresponding to the three-dimensional model.

[0122] In one implementation, the operation execution unit 605 obtains the operation prompt information in the following manner: inputting the input data into the target model to obtain the operation prompt information output by the target model; wherein the target model is optimized using the operation text and the scene semantic information.

[0123] In one implementation, the operation execution unit 605 obtains the operation prompt information in the following manner: extracting operation key information from the operation text; obtaining second prompt information based on the operation key information, the scene semantic information and the input data; and inputting the second prompt information into the target model to obtain the operation prompt information output by the target model.

[0124] In one implementation, the device in this embodiment further includes the following units: Figure 8 As shown in:

[0125] The semantic processing unit 606 is used to update the scene semantic information according to the input data and the operation prompt information.

[0126] The semantic processing unit 606 is further used to obtain the scene semantic information by:

[0127] Obtain scanned images of the three-dimensional scene at multiple viewing angles; perform semantic analysis on visual content in the scanned images to obtain scene semantic information; wherein the scene semantic information includes at least one of the following:

[0128] Scene description information of the three-dimensional scene;

[0129] Object description information of objects included in the three-dimensional scene;

[0130] Three-dimensional coordinate information of objects contained in the three-dimensional scene;

[0131] The relative position relationship between the objects included in the three-dimensional scene.

[0132] In one implementation, the semantic processing unit 606 is also used to: if the current viewing angle information of the three-dimensional scene is different from the viewing angle information corresponding to the scanned image, perform an image scan on the three-dimensional scene under the current viewing angle information to obtain a new scanned image; and update the scene semantic information based on the new scanned image.

[0133] It should be noted that the specific implementation method of each unit in this embodiment can refer to the specific content in the previous text and will not be described in detail here.

[0134] refer to Fig. 9, is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may be an electronic device capable of data processing and image rendering, such as a mobile phone, a tablet device, a notebook or a server. The technical solution in this embodiment is mainly used to reduce the complexity of the user's operation of the three-dimensional scene and improve the user experience.

[0135] Specifically, the electronic device in this embodiment may include at least one processor 901 and a memory 902 connected to the processor 901, wherein:

[0136] The memory 902 is used to store computer programs;

[0137] The processor 901 is used to execute the computer program so that the electronic device can implement the following processing:

[0138] Obtaining input data for a three-dimensional scene, wherein the input data includes information corresponding to a target object;

[0139] Obtaining target position information according to scene semantic information of the three-dimensional scene, the input data and the target model, wherein the target position information represents the position of the target object in the three-dimensional scene, and the target model can generate an output result according to the received natural language instruction;

[0140] Determining target viewing angle information according to the target position information;

[0141] A target image corresponding to the target viewing angle information is rendered using the three-dimensional model corresponding to the three-dimensional scene.

[0142] It can be seen that in an electronic device provided in an embodiment of the present application, a target image at the location of a target object can be rendered based on the scene semantic information, input data and target model of the three-dimensional scene. For the user, only the information corresponding to the target object needs to be input, and the corresponding target image can be rendered without the user dragging the three-dimensional scene to find the target object. This can reduce the complexity of the user's operation of the three-dimensional model and improve the user experience.

[0143] The present application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement any one of the above-mentioned data processing methods.

[0144] The present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the data processing methods described above.

[0145] Taking the target model as a metaverse construction platform as an example, the technical solution of this application is illustrated below:

[0146] First, the current usage of 3D scenes is to display the rendering of the scene representation layer. Users need to know in advance how to use the Metaverse Construction Platform and the functions provided by the Metaverse Construction Platform for 3D scenes. Users need to rely on their own exploration to roam the scene, create mission points, etc. As a result, the problem is that the functions of 3D scenes are diverse, the steps are complicated, and there is a lack of corresponding semantic information and usage prompts. Most ordinary users are not clear about the objects that can be viewed in the 3D scene or the specific steps to achieve user needs, so they use the functions of the 3D scene incorrectly, affecting the user experience.

[0147] In view of the above problems, the technical solution of this application proposes the following key technical points:

[0148] 1. The present application can be used to generate original scanned images of three-dimensional scenes, and input the scanned images (i.e., the scanned images mentioned above) into the visual semantic model to generate detailed scene semantic information of the three-dimensional scene, including scene description information, object description information, three-dimensional coordinate information of the objects, relative position relationships between objects, etc.

[0149] 2. This application can classify the usage and functions of the Metaverse Construction Platform and extract key information, which can be used as prompts later.

[0150] 3. In the present application, the scene semantic information of the three-dimensional scene can be used as the context input of the prompt to the LLM, the LLM can be optimized, and a corresponding agent can be generated based on the LLM with the ability to understand 3D space and three-dimensional scene functions, that is, the agent that implements the data processing method in the present application.

[0151] 4. This application uses scene semantic information such as the position coordinates of objects in the three-dimensional scene as the context of the user's input data, which can enable the agent to accurately locate objects, describe scene content, locate task areas, etc. in the metaverse construction platform.

[0152] 5. When using a 3D scene, users can ask questions or tasks to the agent. The agent calls the LLM, and the LLM gives prompts and solution steps based on its understanding of the 3D scene and returns them to the agent. The agent performs corresponding operations according to the solution steps.

[0153] The following combination Fig.10 The implementation process shown is as follows:

[0154] (1) The visual semantics model performs image captioning on the scanned original image, converting the visual content in the scanned original image into text described in natural language, and at the same time generates scene semantic information containing the 3D coordinates of all objects in the image;

[0155] (2) using the method and function of the Metaverse construction platform and the three-dimensional scene semantic information obtained in step (1) to optimize the LLM, so that the LLM can understand and learn the method and function of the Metaverse construction platform and the three-dimensional scene semantic information obtained in step (1);

[0156] (3) After the above steps (1) and (2), the agent can acquire the ability to describe the three-dimensional scene, the ability to understand 3D, the ability to locate objects, and the ability to execute tasks by calling LLM;

[0157] (4) Ask a question, such as: "Please confirm whether there is a display in the scene?".

[0158] (5) The agent finds the display keywords and the three-dimensional coordinate position of the display from the scene semantic information;

[0159] (6) Based on the three-dimensional coordinates of the display, the agent rotates the virtual camera angle of the three-dimensional scene to illuminate the display position, and answers the user: there is a display in the three-dimensional scene, centered on the screen;

[0160] Furthermore, the user can continue to ask the platform function: "What should I do if I want to place a video material on the display?" Based on this, the agent program gives the execution steps according to the learned information.

[0161] In addition, in this embodiment, through dialogue learning and accumulation of semantic information, the agent can give a more accurate and detailed description, thereby achieving the effect of intelligent interaction and improving user experience.

[0162] In summary, the technical solution of this application has the following advantages:

[0163] 1. Use the semantic information of the 3D scene to improve the ability to understand user instructions.

[0164] 2. Use the usage prompts of 3D scenes to improve user experience.

[0165] 3. Utilize the interactive capabilities of 3D scenes to improve the flexibility of using 3D scenes.

[0166] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0167] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0168] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0169] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, comprising: Obtaining input data for a three-dimensional scene, wherein the input data includes information corresponding to a target object; Obtaining target position information according to scene semantic information of the three-dimensional scene, the input data and the target model, wherein the target position information represents the position of the target object in the three-dimensional scene, and the target model can generate an output result according to the received natural language instruction; Determining target viewing angle information according to the target position information; A target image corresponding to the target viewing angle information is rendered using the three-dimensional model corresponding to the three-dimensional scene.

2. The method according to claim 1, obtaining target position information according to scene semantic information of the three-dimensional scene, the input data and the target model, comprising: Inputting the input data into a target model to obtain target position information output by the target model; Wherein, the target model is optimized using scene semantic information of the three-dimensional scene.

3. The method according to claim 1, obtaining target position information according to scene semantic information of the three-dimensional scene, the input data and the target model, comprising: Obtaining first prompt information according to the scene semantic information of the three-dimensional scene and the input data; The first prompt information is input into a target model to obtain target position information output by the target model.

4. According to the method of claim 1, 2 or 3, after rendering a target image corresponding to the target viewing angle information by using the three-dimensional model corresponding to the three-dimensional scene, the method further comprises: According to the operation prompt information, perform corresponding operations on the target object; The operation prompt information is obtained at least according to the operation text of the three-dimensional model, the input data and the target model, and the operation text represents the operation process corresponding to the three-dimensional model.

5. According to the method of claim 4, the operation prompt information is obtained by: Inputting the input data into the target model to obtain operation prompt information output by the target model; in, The target model is optimized using the operation text and the scene semantic information.

6. According to the method of claim 4, the operation prompt information is obtained by: Extracting key operation information from the operation text; Obtaining second prompt information according to the key operation information, the scene semantic information and the input data; The second prompt information is input into the target model to obtain operation prompt information output by the target model.

7. The method according to claim 4, further comprising: The scene semantic information is updated according to the input data and the operation prompt information.

8. According to the method of claim 1, 2 or 3, the scene semantic information is obtained by: Obtaining scanned images of the three-dimensional scene at multiple viewing angles; Performing semantic analysis on the visual content in the scanned image to obtain scene semantic information; in, The scene semantic information includes at least one of the following: Scene description information of the three-dimensional scene; Object description information of objects included in the three-dimensional scene; Three-dimensional coordinate information of objects contained in the three-dimensional scene; The relative position relationship between the objects included in the three-dimensional scene.

9. The method according to claim 8, further comprising: If the current viewing angle information of the three-dimensional scene is different from the viewing angle information corresponding to the scanned image, performing image scanning on the three-dimensional scene under the current viewing angle information to obtain a new scanned image; The scene semantic information is updated according to the new scanned image.

10. A data processing device, comprising: A data acquisition unit, used to obtain input data for a three-dimensional scene, wherein the input data includes information corresponding to a target object; a position obtaining unit, configured to obtain target position information according to scene semantic information of the three-dimensional scene, the input data and a target model, wherein the target position information represents the position of the target object in the three-dimensional scene, and the target model can generate an output result according to a received natural language instruction; A viewing angle obtaining unit, used to determine target viewing angle information according to the target position information; An image rendering unit is used to render a target image corresponding to the target viewing angle information using a three-dimensional model corresponding to the three-dimensional scene.

Images