Man-machine interaction method and device, medium and electronic equipment

Abstract

The invention discloses a man-machine interaction method and device, a medium and electronic equipment, and the method comprises the steps: responding to a selection instruction of a user for a target object, and displaying a digital 3D model of the target object; under the condition that the display duration of the digital 3D model reaches a preset duration, obtaining an introduction text of the target object; according to the introduction text, reasoning the digital 3D model through a multi-modal large model to obtain a segmentation region related to the introduction text, the multi-modal large model being used for aligning and segmenting the introduction text and the 2D images of the multiple viewing angles, and the 2D images of the multiple viewing angles being obtained by rendering different predefined viewpoints of the digital 3D model; and playing an audio corresponding to the introduction text, and displaying the segmented region. Therefore, by adopting the embodiment of the invention, when a user faces digital display of cultural heritage, abstract text interpretation can be effectively associated with specific cultural details, so that the user experience and learning effect are improved.

Description

Technical Field

[0001] The present application relates to the technical field of mixed reality and human-computer interaction, and in particular to a human-computer interaction method, device, medium and electronic device. Background Art

[0002] With the rapid development of information technology, human-computer interaction technology has made significant progress, especially in the fields of 3D display and virtual reality. These technologies provide a new way for users to interact with digital content, greatly enriching the application scenarios in education, entertainment, health care, tourism and other fields. In particular, the digital preservation and dissemination of cultural heritage has become an area of ​​great concern.

[0003] At present, through digital means, people can appreciate and access cultural heritage in an immersive way, which not only provides a new way to protect and inherit cultural heritage, but also provides the public with a richer and more intuitive experience. Since cultural heritage usually contains profound historical and cultural knowledge, its complexity often exceeds the understanding ability of ordinary users without professional background. As a result, when users face the digital display of cultural heritage, it is difficult to effectively connect abstract textual interpretations with specific cultural details, thus affecting the user's experience and learning effect. Summary of the invention

[0004] The embodiments of the present application provide a human-computer interaction method, device, medium and electronic device. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a human-computer interaction method, the method comprising:

[0006] In response to a user's selection instruction for the target object, displaying a digital 3D model of the target object;

[0007] When the display time of the digital 3D model reaches a preset time, obtaining an introduction text of the target object;

[0008] According to the introduction text, the digital 3D model is inferred through the multimodal large model to obtain the segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with the 2D images of multiple perspectives. The 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints;

[0009] The audio corresponding to the introductory text is played and the segmented area is displayed.

[0010] Optionally, the method further includes:

[0011] Receiving posture information sent by a sensing device, where the posture information is sensed by the sensing device when a user operates a pre-marked three-dimensional prototype of a target object and is calculated based on the sensed spatial information, where the three-dimensional prototype is a physical object manufactured based on a digital 3D model;

[0012] According to the posture information, the display direction of the displayed digital 3D model is updated to keep the posture of the digital 3D model consistent with the posture of the three-dimensional prototype.

[0013] Optionally, based on the introduction text, the digital 3D model is inferred through the multimodal large model to obtain segmented areas related to the introduction text, including:

[0014] Rendering different predefined viewpoints of the digital 3D model to generate 2D images from multiple perspectives;

[0015] Get the text instruction hints present in the introduction text;

[0016] Input 2D images of multiple viewpoints and text instruction prompts into a multimodal large model, and output multiple target 2D regions described by the text instruction prompts;

[0017] Fusion of multiple target 2D regions into a 3D mask;

[0018] Preprocess the 3D mask to obtain segmented regions related to the introductory text.

[0019] Optionally, the multimodal large model includes an alignment module and a segmentation module;

[0020] Input 2D images from multiple perspectives and text instruction prompts into the multimodal large model, and output multiple target 2D areas described by the text instruction prompts, including:

[0021] The alignment module receives 2D images and text instruction prompts from multiple perspectives, extracts visual features from the 2D images, extracts keywords and phrases from the text instruction prompts, uses deep learning technology to align the visual features with the text features, and outputs multiple aligned feature maps;

[0022] The segmentation module receives multiple feature maps output by the alignment module and 2D images of multiple perspectives, performs region segmentation on the 2D image of each perspective based on the multiple feature maps, and outputs multiple target 2D regions described by text instruction prompts.

[0023] Optionally, pre-process the 3D mask to obtain segmented regions related to the introductory text, including:

[0024] Refine the 3D mask;

[0025] Eliminate discontinuities caused by viewpoint changes in the refined 3D mask;

[0026] Filter out low confidence regions for the 3D mask that eliminates discontinuities caused by viewpoint changes;

[0027] The 3D mask that filters out low-confidence areas is projected onto the digital 3D model to obtain the segmented areas related to the introduction text.

[0028] Optionally, Gaussian geodesic reweighting algorithm is used for refinement, visibility smoothing technique is used to eliminate discontinuities caused by viewpoint changes, global filtering strategy is used to filter out low confidence areas, and audio is obtained by converting the introduction text through Text2Speech algorithm.

[0029] Optionally, a digital 3D model of the target object is generated according to the following steps, including:

[0030] Collecting object information of the target object, the object information includes geometric data, image information and on-site measurement information;

[0031] Construct point cloud data of the target object based on geometric data, image information and on-site measurement information;

[0032] Convert point cloud data into a network model to obtain the initial 3D structure;

[0033] The initial 3D structure is subjected to three-dimensional modeling and texture mapping to obtain a digital 3D model of the target object.

[0034] In a second aspect, an embodiment of the present application provides a human-computer interaction device, the device comprising:

[0035] a display module, for displaying a digital 3D model of a target object in response to a selection instruction for the target object;

[0036] An acquisition module, used to acquire the introduction text of the target object when the display time of the digital 3D model reaches a preset time;

[0037] An inference module is used to infer the digital 3D model through the multimodal large model according to the introduction text to obtain the segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with the 2D images of multiple perspectives. The 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints;

[0038] The playing and display module is used to play the audio corresponding to the introduction text and display the segmented area.

[0039] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0040] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0041] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0042] In the embodiment of the present application, according to the introduction text, the digital 3D model is inferred through a multimodal large model to obtain a segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives, and the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints. This process involves accurately aligning and segmenting the introduction text with a set of 2D images rendered from the digital 3D model at multiple predefined viewpoints. In this way, the abstract text interpretation is effectively linked to the specific cultural details, which greatly enhances the user's immersion and learning depth, thereby significantly improving the user experience and educational effectiveness.

[0043] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1 It is a flowchart of a human-computer interaction method provided in an embodiment of the present application;

[0046] Figure 2 It is a schematic diagram of a three-dimensional mask fusion process provided by this application;

[0047] Figure 3 It is a schematic diagram of segmented regions related to the introduction text provided by the present application;

[0048] Figure 4 It is a schematic diagram of a three-dimensional prototype generation process provided in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of an interaction of a user operating a three-dimensional prototype of a target object provided by an embodiment of the present application;

[0050] Figure 6It is a schematic diagram of the overall architecture of a human-computer interaction provided by an embodiment of the present application;

[0051] Figure 7 It is a structural schematic diagram of a human-computer interaction device provided by the present application;

[0052] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0054] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0055] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0056] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0057] The present application provides a human-computer interaction method, device, medium and electronic device to solve the problems existing in the above-mentioned related technical issues. In an embodiment of the present application, according to the introduction text, the digital 3D model is inferred through a multimodal large model, and a segmented area related to the introduction text can be obtained. The multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives. The 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints. This process involves accurately aligning and segmenting the introduction text with a set of 2D images rendered from the digital 3D model at multiple predefined viewpoints. In this way, the abstract text interpretation is effectively linked to the specific cultural details, which greatly enhances the user's immersion and learning depth, thereby significantly improving the user experience and educational effectiveness. The following is a detailed description using an exemplary embodiment.

[0058] The following will be combined with the attached Figure 1 -Attached Figure 6 , the human-computer interaction method provided by the embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a human-computer interaction device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0059] See also Figure 1 , which is a flow chart of a human-computer interaction method provided in an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0060] S101, in response to a user's selection instruction for a target object, displaying a digital 3D model of the target object;

[0061] The user refers to an individual who interacts with the system or device, and can be a learner of cultural heritage. The target object is the specific cultural heritage selected or focused on by the user. The selection instruction is generated by the user selecting or interacting with the target object, which can be a click, touch, voice command or other form of user input. A digital 3D model is a virtual three-dimensional object created by digital technology.

[0062] In some embodiments, the user selects the cultural heritage they are interested in, such as a porcelain, by clicking on a touch screen, using a mouse, or issuing a selection instruction through a graphical interface (GUI). The system receives the user's selection instruction and converts it into a query for the corresponding cultural heritage in the database. Based on the user's selection, the system retrieves the digital 3D model associated with the selected cultural heritage. Using 3D rendering technology, the retrieved digital 3D model is rendered and displayed on the screen with high resolution and realism.

[0063] The database contains pre-built mapping relationships between logos and digital 3D models.

[0064] In an embodiment of the present application, a digital 3D model of a target object is generated according to the following steps, including: collecting object information of the target object, the object information including geometric data, image information and on-site measurement information; constructing point cloud data of the target object based on the geometric data, image information and on-site measurement information; converting the point cloud data into a network model to obtain an initial 3D structure; performing three-dimensional modeling and texture mapping on the initial 3D structure to obtain a digital 3D model of the target object.

[0065] Furthermore, after obtaining the digital 3D model of each target object, an identifier of each target object may be created, and then the identifier of each target object and the digital 3D model of each target object may be stored to obtain a pre-constructed mapping relationship between the identifier and the digital 3D model.

[0066] S102, when the display time of the digital 3D model reaches a preset time, obtaining an introduction text of the target object;

[0067] The display duration refers to the length of time the digital 3D model is displayed on the user interface. The preset duration is a time limit set in advance to determine whether to obtain the introduction text of the target object. The introduction text is textual information that provides background information, historical significance, artistic value, etc. for the target object.

[0068] In some embodiments, when a user selects a cultural heritage object, the system displays the corresponding digital 3D model and starts timing. The system continuously monitors the duration of the currently displayed 3D model. Once the preset duration is reached, the timer triggers an event. After the event is triggered, the system automatically retrieves the introductory text associated with the displayed cultural heritage object from the database.

[0069] In the embodiment of the present application, when the display time of the digital 3D model does not reach the preset time, the timing continues.

[0070] S103, according to the introduction text, the digital 3D model is inferred by using a multimodal large model to obtain a segmented area related to the introduction text, the multimodal large model is used to align and segment the introduction text with the 2D images of multiple perspectives, the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints;

[0071] Among them, the segmented area is a specific part divided from the overall image or model, which is associated with specific information or features. A 2D image is a two-dimensional image, that is, an image on a plane. Compared with a three-dimensional model, it only contains width and height information. The viewing angle is a specific position and direction when observing or capturing an image. Different viewing angles can provide different visual information. A predefined viewpoint is a specific position and direction set in advance for observing or rendering a 3D model during the 3D modeling and rendering process.

[0072] In some embodiments of the present application, the specific process of inferring the digital 3D model through the multimodal large model based on the introduction text to obtain the segmented area related to the introduction text includes: rendering different predefined viewpoints of the digital 3D model to generate 2D images of multiple perspectives; obtaining text instruction prompts in the introduction text; inputting the 2D images of multiple perspectives and the text instruction prompts into the multimodal large model, and outputting multiple target 2D areas described by the text instruction prompts; fusing the multiple target 2D areas into a three-dimensional mask; and preprocessing the three-dimensional mask to obtain the segmented area related to the introduction text.

[0073] The text instruction prompt is the instruction or prompt contained in the introduction text, which is used to guide the user to learn relevant knowledge of the target object. The three-dimensional mask is to fuse multiple 2D segmented areas to form a three-dimensional mask that can be applied to the 3D model to identify and distinguish different parts of the model.

[0074] In an embodiment of the present application, by rendering a digital 3D model from multiple predefined viewpoints to generate a 2D image, combined with the instruction prompts in the introduction text, and using a multimodal large model for in-depth analysis and reasoning, it is possible to accurately identify and segment the 2D target area that is closely related to the text description. These areas are further fused to form a three-dimensional mask, and the mask is preprocessed to finally obtain a segmented area that accurately corresponds to the content of the introduction text. This method not only enhances the user's understanding and perception of the 3D model, but also provides a more intuitive and interactive learning experience by combining abstract text with specific visual elements. It is particularly suitable for the digital display and education of cultural heritage, greatly improving the efficiency of information transmission.

[0075] Among them, the multimodal large model includes an alignment module and a segmentation module.

[0076] In some embodiments of the present application, the specific process of inputting 2D images from multiple perspectives and text instruction prompts into a multimodal large model and outputting multiple target 2D areas described by the text instruction prompts includes: an alignment module receives 2D images from multiple perspectives and text instruction prompts, extracts visual features from the 2D images, extracts keywords and phrases from the text instruction prompts, uses deep learning technology to align the visual features with the text features, and outputs multiple aligned feature maps; a segmentation module receives multiple feature maps output by the alignment module and 2D images from multiple perspectives, performs region segmentation on the 2D image from each perspective based on the multiple feature maps, and outputs multiple target 2D areas described by the text instruction prompts.

[0077] In some embodiments of the present application, the specific process of preprocessing the three-dimensional mask to obtain the segmented area related to the introduction text includes: refining the three-dimensional mask; eliminating the discontinuity caused by the viewpoint change on the refined three-dimensional mask; filtering out the low-confidence area on the three-dimensional mask that eliminates the discontinuity caused by the viewpoint change; and projecting the three-dimensional mask that filters out the low-confidence area onto the digital 3D model to obtain the segmented area related to the introduction text.

[0078] Specifically, Gaussian geodesic reweighting algorithm is used for refinement, visibility smoothing technique is used to eliminate discontinuities caused by viewpoint changes, global filtering strategy is used to filter out low confidence areas, and audio is obtained by converting the introduction text through Text2Speech algorithm.

[0079] For example Figure 2 As shown, the target object is a human-shaped cultural heritage. First, the digital 3D model of the human-shaped cultural heritage is rendered from different predefined viewpoints to generate multi-view images. Then, the 2D images of multiple viewpoints and text instruction prompts are input into the multimodal large model to output multiple target 2D regions described by the text instruction prompts. The multiple target 2D regions are fused into a three-dimensional mask through a fusion module. The three-dimensional mask is preprocessed and projected onto the digital 3D model of the human-shaped cultural heritage, and the image can be obtained, for example Figure 3 The segmented area related to the introductory text.

[0080] S104, playing the audio corresponding to the introduction text and displaying the segmented areas.

[0081] In some embodiments of the present application, after the segmented regions related to the introduction text are obtained, the introduction text can be converted into audio through the Text2Speech algorithm, and the audio can be played, and the segmented regions can be displayed.

[0082] In the embodiment of the present application, according to the introduction text, the digital 3D model is inferred through a multimodal large model to obtain a segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives, and the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints. This process involves accurately aligning and segmenting the introduction text with a set of 2D images rendered from the digital 3D model at multiple predefined viewpoints. In this way, the abstract text interpretation is effectively linked to the specific cultural details, which greatly enhances the user's immersion and learning depth, thereby significantly improving the user experience and educational effectiveness.

[0083] In an embodiment of the present application, after displaying the digital 3D model of the target object, the user can operate the pre-marked three-dimensional prototype of the target object to control the display direction of the digital 3D model; or after playing the audio corresponding to the introduction text and displaying the segmented area, the user can operate the pre-marked three-dimensional prototype of the target object to control the display direction of the digital 3D model. The user's operation of the pre-marked three-dimensional prototype of the target object can be determined by the user, and based on the user's operation at different times, the system can control the display direction of the digital 3D model at different times.

[0084] Specifically, the system receives posture information sent by a perception device, which is perceived by the perception device when a user operates a pre-marked three-dimensional prototype of a target object and calculated based on the perceived spatial information, and the three-dimensional prototype is a physical object manufactured based on the digital 3D model; according to the posture information, the display direction of the displayed digital 3D model is updated to keep the posture of the digital 3D model consistent with that of the three-dimensional prototype.

[0085] Among them, sensing devices refer to devices that can detect and identify target objects in the environment, such as cameras, motion trackers, etc. Posture information is data that describes the direction and position of an object in space, including rotation angle, tilt, translation, etc. Physical objects refer to physical objects in the real world, as opposed to digital models. Display direction is the direction that the digital 3D model faces on the screen or in a virtual reality environment.

[0086] In this case, the 3D prototype is created by a 3D printer and the digital 3D model is created in a 3D mesh. Figure 4 As shown, the digital 3D model in the three-dimensional grid is printed by 3D printing technology to obtain a three-dimensional basic prototype, and then the three-dimensional basic prototype is colored to obtain a three-dimensional prototype.

[0087] After obtaining the 3D prototype, it is glued to an acrylic board with ARUco marks on the four corners. Each mark has a unique structure and serves as a key input to the attitude estimation algorithm. The camera captures the spatial information of these marks, and then processes this information through the attitude estimation algorithm to determine the direction of the acrylic board, and then determine the direction of the prototype, and obtain the attitude information of the 3D prototype. The camera finally sends the attitude information to the system.

[0088] For example Figure 5 As shown, the user operates the 3D prototype of the target object, and the camera captures the posture information of the 3D prototype and sends it to the system in the holographic cabin. The system updates the display direction of the displayed digital 3D model based on the posture information to keep the posture of the digital 3D model consistent with that of the 3D prototype. Finally, the user can observe the updated display direction of the digital 3D model fed back by the holographic cabin.

[0089] In an embodiment of the present application, by receiving posture information sent by a sensing device, the system can accurately capture the spatial dynamics of a user operating a pre-marked three-dimensional prototype. This information is based on real-time monitoring and calculation of user behavior by the sensing device, ensuring synchronization between the digital 3D model and the actual object. Updating the display direction of the digital 3D model to be consistent with the posture of the three-dimensional prototype not only enhances the intuitiveness and accuracy of user interaction, but also provides an immersive experience, allowing users to interact with virtual content more naturally. This method has significant advantages in many fields such as education, exhibitions, and design verification. Especially in the digital display of cultural heritage, it allows users to explore and learn in a whole new way, greatly improving the user experience and educational effect.

[0090] For example Figure 6 As shown, Figure 6 This is a schematic diagram of the overall architecture of human-computer interaction provided by the present application. By reconstructing the cultural heritage in three dimensions, a digital 3D model of the cultural heritage can be obtained, and the digital 3D model can be input into a holographic cabin for display. At the same time, a text introduction of the cultural heritage and the digital 3D model are input into a multimodal large model for processing, and a segmented area related to the introduction text can be obtained. The segmented area related to the introduction text is sent to the holographic cabin for display, and the text introduction is converted into audio for playback. The user can control the display angle of the digital 3D model of the cultural heritage by operating the 3D printed prototype.

[0091] In the embodiment of the present application, according to the introduction text, the digital 3D model is inferred through a multimodal large model to obtain a segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives, and the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints. This process involves accurately aligning and segmenting the introduction text with a set of 2D images rendered from the digital 3D model at multiple predefined viewpoints. In this way, the abstract text interpretation is effectively linked to the specific cultural details, which greatly enhances the user's immersion and learning depth, thereby significantly improving the user experience and educational effectiveness.

[0092] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0093] See also Figure 7 , which shows a schematic diagram of the structure of a human-computer interaction device provided by an exemplary embodiment of the present application. The human-computer interaction device can be implemented as all or part of an electronic device through software, hardware or a combination of both. The device 1 includes a display module 10, an acquisition module 20, an inference module 30, and a playback and display module 40.

[0094] A display module 10, for displaying a digital 3D model of a target object in response to a selection instruction for the target object;

[0095] An acquisition module 20 is used to acquire the introduction text of the target object when the display time of the digital 3D model reaches a preset time;

[0096] The inference module 30 is used to infer the digital 3D model through the multimodal large model according to the introduction text to obtain the segmented area related to the introduction text, and the multimodal large model is used to align and segment the introduction text with the 2D images of multiple perspectives, and the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints;

[0097] The playing and displaying module 40 is used to play the audio corresponding to the introduction text and display the segmented areas.

[0098] It should be noted that the human-computer interaction device provided in the above embodiment only uses the division of the above functional modules as an example when executing the human-computer interaction method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the human-computer interaction device provided in the above embodiment and the human-computer interaction method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0099] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0100] In the embodiment of the present application, according to the introduction text, the digital 3D model is inferred through a multimodal large model to obtain a segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives, and the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints. This process involves accurately aligning and segmenting the introduction text with a set of 2D images rendered from the digital 3D model at multiple predefined viewpoints. In this way, the abstract text interpretation is effectively linked to the specific cultural details, which greatly enhances the user's immersion and learning depth, thereby significantly improving the user experience and educational effectiveness.

[0101] The present application also provides a computer-readable medium having program instructions stored thereon, and when the program instructions are executed by a processor, the human-computer interaction methods provided by the above-mentioned various method embodiments are implemented.

[0102] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the human-computer interaction methods of the above-mentioned various method embodiments.

[0103] See also Figure 8 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0104] The communication bus 1002 is used to realize the connection and communication between these components.

[0105] The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.

[0106] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0107] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts within the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0108] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 8 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a human-computer interaction application program.

[0109] exist Figure 8In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the human-computer interaction application stored in the memory 1005 and specifically perform the following operations:

[0110] In response to a user's selection instruction for the target object, displaying a digital 3D model of the target object;

[0111] When the display time of the digital 3D model reaches a preset time, obtaining an introduction text of the target object;

[0112] According to the introduction text, the digital 3D model is inferred through the multimodal large model to obtain the segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with the 2D images of multiple perspectives. The 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints;

[0113] The audio corresponding to the introductory text is played and the segmented area is displayed.

[0114] In one embodiment, the processor 1001 further performs the following operations:

[0115] Receiving posture information sent by a sensing device, where the posture information is sensed by the sensing device when a user operates a pre-marked three-dimensional prototype of a target object and is calculated based on the sensed spatial information, where the three-dimensional prototype is a physical object manufactured based on a digital 3D model;

[0116] According to the posture information, the display direction of the displayed digital 3D model is updated to keep the posture of the digital 3D model consistent with the posture of the three-dimensional prototype.

[0117] In one embodiment, when the processor 1001 performs reasoning on the digital 3D model through the multimodal large model according to the introduction text to obtain the segmented area related to the introduction text, the following operations are specifically performed:

[0118] Rendering different predefined viewpoints of the digital 3D model to generate 2D images from multiple perspectives;

[0119] Get the text instruction hints present in the introduction text;

[0120] Input 2D images of multiple viewpoints and text instruction prompts into a multimodal large model, and output multiple target 2D regions described by the text instruction prompts;

[0121] Fusion of multiple target 2D regions into a 3D mask;

[0122] Preprocess the 3D mask to obtain segmented regions related to the introductory text.

[0123] In one embodiment, when the processor 1001 inputs the 2D images of multiple perspectives and the text instruction prompts into the multimodal macro model and outputs the multiple target 2D regions described by the text instruction prompts, the processor 1001 specifically performs the following operations:

[0124] The alignment module receives 2D images and text instruction prompts from multiple perspectives, extracts visual features from the 2D images, extracts keywords and phrases from the text instruction prompts, uses deep learning technology to align the visual features with the text features, and outputs multiple aligned feature maps;

[0125] The segmentation module receives multiple feature maps output by the alignment module and 2D images of multiple perspectives, performs region segmentation on the 2D image of each perspective based on the multiple feature maps, and outputs multiple target 2D regions described by text instruction prompts.

[0126] In one embodiment, when the processor 1001 pre-processes the three-dimensional mask to obtain the segmented area related to the introduction text, the processor 1001 specifically performs the following operations:

[0127] Refine the 3D mask;

[0128] Eliminate discontinuities caused by viewpoint changes in the refined 3D mask;

[0129] Filter out low confidence regions for the 3D mask that eliminates discontinuities caused by viewpoint changes;

[0130] The 3D mask that filters out low-confidence areas is projected onto the digital 3D model to obtain the segmented areas related to the introduction text.

[0131] In one embodiment, when the processor 1001 generates a digital 3D model of the target object, the processor 1001 specifically performs the following operations:

[0132] Collect object information of the target object, which includes geometric data, image information and on-site measurement information;

[0133] Construct point cloud data of the target object based on geometric data, image information and on-site measurement information;

[0134] Convert point cloud data into a network model to obtain the initial 3D structure;

[0135] The initial 3D structure is subjected to three-dimensional modeling and texture mapping to obtain a digital 3D model of the target object.

[0136] In the embodiment of the present application, according to the introduction text, the digital 3D model is inferred through a multimodal large model to obtain a segmented area related to the introduction text. The multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives, and the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints. This process involves accurately aligning and segmenting the introduction text with a set of 2D images rendered from the digital 3D model at multiple predefined viewpoints. In this way, the abstract text interpretation is effectively linked to the specific cultural details, which greatly enhances the user's immersion and learning depth, thereby significantly improving the user experience and educational effectiveness.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the human-computer interaction program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium of the human-computer interaction program can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0138] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A human-computer interaction method, characterized in that: The method comprises: In response to a user's selection instruction for a target object, displaying a digital 3D model of the target object; When the display time of the digital 3D model reaches a preset time, obtaining an introduction text of the target object; According to the introduction text, the digital 3D model is inferred by a multimodal large model to obtain a segmented area related to the introduction text, wherein the multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives, wherein the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints; The audio corresponding to the introduction text is played, and the segmented area is displayed.

2. The method according to claim 1, characterized in that The method further comprises: Receiving posture information sent by a sensing device, wherein the posture information is sensed by the sensing device when the user operates a pre-marked three-dimensional prototype of the target object and is calculated based on the sensed spatial information, and the three-dimensional prototype is a physical object manufactured based on the digital 3D model; According to the posture information, the display direction of the displayed digital 3D model is updated to keep the posture of the digital 3D model consistent with the posture of the three-dimensional prototype.

3. The method according to claim 1, characterized in that The method of inferring the digital 3D model through a multimodal large model according to the introduction text to obtain a segmented area related to the introduction text includes: Rendering different predefined viewpoints of the digital 3D model to generate 2D images of multiple viewing angles; Obtaining text instruction prompts existing in the introduction text; Inputting the 2D images of the multiple perspectives and the text instruction prompt into a multimodal macro model, and outputting multiple target 2D regions described by the text instruction prompt; Merging the multiple target 2D regions into a three-dimensional mask; The three-dimensional mask is preprocessed to obtain a segmented area related to the introduction text.

4. The method according to claim 3, characterized in that: The multimodal large model includes an alignment module and a segmentation module; The step of inputting the 2D images of the multiple viewing angles and the text instruction prompt into the multimodal macro model and outputting the multiple target 2D regions described by the text instruction prompt comprises: The alignment module receives 2D images of multiple perspectives and the text instruction prompt, extracts visual features from the 2D images, extracts keywords and phrases from the text instruction prompt, aligns the visual features with the text features using deep learning technology, and outputs a plurality of aligned feature maps; The segmentation module receives the multiple feature maps output by the alignment module and the 2D images of the multiple perspectives, performs region segmentation on the 2D image of each perspective based on the multiple feature maps, and outputs multiple target 2D regions described by the text instruction prompt.

5. The method according to claim 3, characterized in that: The preprocessing of the three-dimensional mask to obtain a segmented area related to the introduction text includes: Refining the three-dimensional mask; Eliminating discontinuity caused by viewpoint change on the refined three-dimensional mask; filtering out low confidence regions from the three-dimensional mask that eliminates discontinuities caused by viewpoint changes; The three-dimensional mask with low confidence areas filtered out is projected onto the digital 3D model to obtain a segmented area related to the introduction text.

6. The method according to claim 5, characterized in that The refinement adopts Gaussian geodesic reweighting algorithm, the elimination of discontinuity caused by viewpoint change adopts visibility smoothing technology, the filtering out of low confidence areas adopts global filtering strategy, and the audio is obtained by converting the introduction text through Text2Speech algorithm.

7. The method according to claim 1, generating a digital 3D model of the target object according to the following steps, comprising: Collecting object information of the target object, wherein the object information includes geometric data, image information, and on-site measurement information; Constructing point cloud data of the target object based on the geometric data, image information and on-site measurement information; Converting the point cloud data into a network model to obtain an initial 3D structure; The initial 3D structure is subjected to three-dimensional modeling and texture mapping to obtain a digital 3D model of the target object.

8. A human-computer interaction device, characterized in that: The device comprises: A display module, for displaying a digital 3D model of a target object in response to a selection instruction for the target object; An acquisition module, configured to acquire the introduction text of the target object when the display time of the digital 3D model reaches a preset time; an inference module, configured to infer the digital 3D model through a multimodal large model according to the introduction text to obtain a segmented area related to the introduction text, wherein the multimodal large model is used to align and segment the introduction text with 2D images of multiple perspectives, wherein the 2D images of multiple perspectives are obtained by rendering the digital 3D model from different predefined viewpoints; The playing and displaying module is used to play the audio corresponding to the introduction text and display the segmented area.

9. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 7.

Images