Real-time object interaction method based on augmented reality and related device

Abstract

The invention discloses a real-time object interaction method based on augmented reality and a related device, and relates to the field of AR, and the method comprises the steps: obtaining a three-dimensional model of a real object in a target image sequence, generating a real-time scene understanding model based on the obtained three-dimensional model, carrying out the multi-mode intention recognition processing of a real-time video sequence of a user, and obtaining a real-time scene understanding model. And obtaining a real-time intention recognition result of the user, obtaining a real-time scene three-dimensional model based on the real-time intention recognition result and the real-time scene understanding model, and performing AR rendering according to the real-time scene three-dimensional model and the target image sequence to obtain an AR rendering result. The real-time scene understanding model can reflect the real-time change of the real object, so that the behavior of the virtual object can change along with the real object, the interaction reality sense is higher, the multi-mode intention recognition processing enables the interaction mode of the virtual object and the real object to be more consistent with the user intention, and the user experience is better.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and in particular to a real-time object interaction method based on augmented reality and related devices. Background Art

[0002] Augmented Reality (AR) technology is a technology that enables users to perceive and interact with the virtual world in the real world by integrating digital information in the virtual world with the real world.

[0003] In traditional AR technology, the rules and animations for the interaction between virtual objects and the real world are usually predefined. On this basis, the user's intention is understood through the user's touch screen operation or simple gesture recognition, and the existing AR physics engine is used to adjust the three-dimensional model of the virtual object based on the user's intention, so as to achieve the purpose of interaction between virtual objects and the real world.

[0004] However, traditional AR technology has the following problems: (1) The predefined interaction rules and animations cannot adjust the behavior of virtual objects in real time according to changes in the real world, resulting in a lack of realism in the interaction between virtual objects and the real world and an inability to achieve accurate interaction; (2) The user's intentions cannot be accurately understood through touch screen operations or simple gesture recognition, resulting in the interaction between virtual objects and the real world not being consistent with the user's intentions, resulting in a poor user experience. Summary of the invention

[0005] In view of the above problems, the present application provides a real-time object interaction method based on augmented reality and related devices to solve the problems existing in the prior art, such as poor realism of the interaction between virtual objects and the real world, low realism and low interaction fluency of the physical effects simulated by the AR physics engine, and the inconsistency between the interaction mode between virtual objects and the real world and the user's intention. The specific scheme is as follows:

[0006] The first aspect of the present application provides a real-time object interaction method based on augmented reality, comprising:

[0007] Acquire a three-dimensional model of a real object in a target image sequence, wherein the target image sequence is obtained by continuously photographing the real world;

[0008] Generate a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence, wherein the real-time scene understanding model includes a scene understanding model corresponding to each image in the target image sequence, and the scene understanding model corresponding to each image can characterize the spatial relationship between the real objects in the image and the distribution of functional areas in the real scene corresponding to the image;

[0009] Performing multimodal intention recognition processing on the user's real-time video sequence to obtain a real-time intention recognition result of the user;

[0010] Based on the real-time intention recognition result and the real-time scene understanding model, a real-time scene three-dimensional model capable of reflecting the interaction mode between the real object and the virtual object is obtained;

[0011] AR rendering is performed according to the real-time scene three-dimensional model and the target image sequence to obtain an AR rendering result.

[0012] In a possible implementation, acquiring a three-dimensional model of a real object in a target image sequence includes:

[0013] Using an improved semantic segmentation model to perform semantic segmentation processing on each image in the target image sequence to obtain a label image corresponding to each image, wherein each pixel in the label image is annotated with a semantic category label, and the improved semantic segmentation model fuses shallow high-resolution features with deep low-resolution features in the encoding stage to obtain fused features as input of the decoding stage, and performs weighted processing on the decoded feature map in the decoding stage to obtain a weighted feature map as the final output of the decoding stage;

[0014] An improved target detection model is used to perform target detection on the label image corresponding to each image to obtain the recognition result of the real object in each image, wherein the improved target detection model performs spatial attention weighting and channel attention weighting on the feature maps output by the Chinese focus layer, the Chinese cross-stage local layer and the feature pyramid network layer, respectively, and uses the fusion value of the spatial attention weighted result and the channel attention weighted result as the input of the next layer;

[0015] Processing the target image sequence to obtain the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real object in the target image sequence;

[0016] Based on the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real object in the target image sequence, an initial model of the real object in each image is obtained;

[0017] Based on the recognition result of the real object in each image, the initial model of the real object in each image is updated to obtain the three-dimensional model of the real object in each image; so as to obtain the three-dimensional model of the real object in the target image sequence.

[0018] In a possible implementation, generating a real-time scene understanding model based on a three-dimensional model of a real object in the target image sequence includes:

[0019] Based on the three-dimensional model of the real object in each image, a target scene graph corresponding to each image is constructed, wherein the target scene graph corresponding to each image has the real object in the image as a node, the attribute features of the real object as a node feature, and the spatial relationship between the real objects in the image as an edge;

[0020] The real-time scene understanding model is generated based on the target scene graphs corresponding to the respective images in the target image sequence.

[0021] In a possible implementation, constructing a target scene graph corresponding to each image based on the three-dimensional model of the real object in each image includes:

[0022] Modeling is performed based on the three-dimensional model of the real object in each image to obtain an initial scene graph corresponding to each image;

[0023] Querying semantically associated objects of the real object in each image from a pre-generated knowledge graph, and generating an embedding vector of the semantically associated objects of the real object in each image;

[0024] The initial scene graph corresponding to each image is updated based on the embedding vector of the semantically associated object of the real object in each image to obtain the target scene graph corresponding to each image.

[0025] In a possible implementation, performing multimodal intention recognition processing on the real-time video sequence of the user to obtain the real-time intention recognition result of the user includes:

[0026] Identify key points of the user's hand in the real-time video sequence to obtain a gesture recognition result of the user in the real-time video sequence;

[0027] Performing speech recognition on the audio data of the user in the real-time video sequence to obtain a speech recognition result of the user in the real-time video sequence;

[0028] Tracking and identifying the eye movement and head posture of the user in the real-time video sequence to obtain a sight line tracking and identification result of the user in the real-time video sequence;

[0029] The real-time intention recognition result is obtained based on the gesture recognition result, the voice recognition result and the sight tracking recognition result.

[0030] In a possible implementation, performing AR rendering according to the real-time scene three-dimensional model and the target image sequence includes:

[0031] Extracting depth information of each image in the target image sequence to obtain a sparse depth map corresponding to each image;

[0032] Performing dense depth prediction on each image in the target image sequence to obtain a dense depth map corresponding to each image;

[0033] Performing weighted fusion on the sparse depth map and the dense depth map corresponding to each image to obtain a fused depth map corresponding to each image;

[0034] Based on the fused depth maps respectively corresponding to the images in the target image sequence, the real-time scene three-dimensional model and the target image sequence, generating a light and shadow rendering image sequence corresponding to the target image sequence;

[0035] Performing virtual-real fusion processing based on the light and shadow rendering image sequence and the fused depth map corresponding to each image to obtain a first augmented reality rendering image sequence, wherein the virtual-real fusion processing includes virtual-real occlusion processing, smoothing processing of virtual object edges and semi-transparent areas, and feathering processing of virtual object edges;

[0036] The first augmented reality rendering image sequence is subjected to preset post-processing to obtain a second augmented reality rendering image sequence with a stronger sense of reality, wherein the preset post-processing includes at least one of the following processing: super-resolution reconstruction, color correction processing, anti-aliasing processing, global illumination processing, blur adjustment processing and preset lens information adding processing.

[0037] A second aspect of the present application provides a real-time object interaction device based on augmented reality, comprising:

[0038] A model acquisition module, used to acquire a three-dimensional model of a real object in a target image sequence, wherein the target image sequence is obtained by continuously photographing the real world;

[0039] A first model generation module is used to generate a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence, wherein the real-time scene understanding model includes a scene understanding model corresponding to each image in the target image sequence, and the scene understanding model corresponding to each image can characterize the spatial relationship between the real objects in the image and the distribution of functional areas in the real scene corresponding to the image;

[0040] An intention recognition module is used to perform multimodal intention recognition processing on a user's real-time video sequence to obtain a real-time intention recognition result of the user;

[0041] A second model generation module is used to obtain a real-time scene three-dimensional model that can reflect the interaction mode between the real object and the virtual object based on the real-time intention recognition result and the real-time scene understanding model;

[0042] The AR rendering module is used to perform AR rendering according to the real-time scene three-dimensional model and the target image sequence to obtain an AR rendering result.

[0043] A third aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the real-time object interaction method based on augmented reality according to the first aspect or any implementation of the first aspect.

[0044] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0045] The memory is used to store computer programs;

[0046] The processor is used to execute the computer program so that the electronic device can implement the real-time object interaction method based on augmented reality according to the above-mentioned first aspect or any implementation manner of the first aspect.

[0047] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the real-time object interaction method based on augmented reality according to the above-mentioned first aspect or any implementation of the first aspect.

[0048] By means of the above technical solution, the present application provides a real-time object interaction method based on augmented reality, which can generate a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence, and the real-time scene understanding model can characterize the spatial distribution of the real object and / or the distribution of the functional area in the real world at each shooting moment of the target image sequence. Therefore, when the spatial relationship of the real objects in the real world photographed changes, it can be reflected in the real-time scene understanding model in real time, and then the real-time scene understanding model is combined with the real-time intention recognition result of the user, so that the behavior of the virtual object can change with the change of the real object, and the interaction between the virtual object and the real object is more realistic. The real-time intention recognition result of the user is obtained based on multimodal consciousness recognition processing. Through multimodal consciousness recognition, the real-time intention of the user can be identified more accurately, and then the interaction mode between the virtual object and the real object is more consistent with the user's intention, which improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0050] Figure 1 A schematic diagram of a system architecture provided for this application;

[0051] Figure 2 A schematic diagram of an optional hardware structure of a terminal 100 provided in this application;

[0052] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0053] Figure 4 A flowchart of a real-time object interaction method based on augmented reality provided in this application;

[0054] Figure 5 A schematic diagram of the structure of a real-time object interaction device based on augmented reality provided in this application;

[0055] Figure 6 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0056] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0057] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0058] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0059] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0060] Among them, an application corresponding to the third-party system can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0061] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0062] Next describe Figure 1 The product form of the mid-terminal 100;

[0063] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0064] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0065] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0066] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0067] Among them, the input device 132 can receive input data and the like.

[0068] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0069] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0070] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0071] Among them, the memory 120 can be used to store software codes related to the real-time object interaction method based on augmented reality, the processor 170 can execute the steps of the real-time object interaction method based on augmented reality, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0072] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile Communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0073] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0074] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0075] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0076] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0077] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0078] Next describe Figure 1 The product form of the server 200;

[0079] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0080] The bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0081] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0082] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0083] Among them, the memory 204 can be used to store software codes related to the real-time object interaction method based on augmented reality, the processor 202 can execute the steps of the real-time object interaction method based on augmented reality of the chip, and can also schedule other units to implement corresponding functions.

[0084] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, DSPs, microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0085] The embodiment of the present application provides a real-time object interaction method based on augmented reality. The real-time object interaction method based on augmented reality in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0086] Reference Figure 4 , Figure 4 A flowchart of a real-time object interaction method based on augmented reality is provided in an embodiment of the present application, such as Figure 4 As shown, the method may include:

[0087] Step S401: Acquire a three-dimensional model of a real object in a target image sequence.

[0088] The target image sequence is obtained by continuously photographing the real world, and the three-dimensional model of the real object in the target image sequence includes at least one three-dimensional model of the real object contained in each image in the target image sequence.

[0089] In augmented reality (AR) technology, in order to accurately overlay or integrate virtual objects into the real world in a more natural and realistic way, it is usually necessary to model the real world, or more precisely, to model real objects in the real world.

[0090] In a possible implementation, the real world can be continuously photographed by a camera to obtain a target image sequence, and then a three-dimensional model of the real object contained in the target image sequence is modeled based on the target image sequence.

[0091] Optionally, the camera is an RGB-D camera, that is, an RGB camera with depth information, and the target image sequence is a color image sequence. Of course, the camera can also be of other types, which is not specifically limited in this application.

[0092] Optionally, the camera's acquisition frequency is 30 frames per second. This setting can ensure that the application can process dynamically changing scenes in real time.

[0093] It should be noted that the above-mentioned "dynamically changing scene" refers to the change of the real world presented on the display object, but it does not necessarily mean that the real world has actually changed. For example, the movement of the camera causes the scene presented on the camera display to change, but the real world does not change during this process.

[0094] Step S402: Generate a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence.

[0095] Among them, the real-time scene understanding model includes a scene understanding model corresponding to each image in the target image sequence, and the scene understanding model corresponding to each image can characterize the spatial relationship between real objects in the image and the distribution of functional areas in the real scene corresponding to the image.

[0096] Specifically, although the three-dimensional model of the real object in the target image sequence obtained in the previous step can be used to determine the specific position of the real object, the three-dimensional models of each real object are relatively isolated and lack connection with each other. In order to be able to more accurately superimpose or fuse virtual objects, it is necessary to first establish a connection between the three-dimensional models of the real object obtained in the previous step, that is, it is necessary to enhance the understanding of the scene where the real object is located, so as to accurately superimpose or fuse virtual objects based on the understood model.

[0097] In this embodiment, a real-time scene understanding model can be generated based on the three-dimensional model of the real object in the target image sequence, and the real-time scene understanding model includes a scene understanding model corresponding to each image in the target image sequence. The scene understanding model corresponding to each image can characterize the spatial relationship between the real objects contained in the image, for example, real object a is above real object b, real object c is close to real object d, and so on; the scene understanding model corresponding to each image also includes the distribution of functional areas in the real scene corresponding to each image, for example, where the work area in the real scene is located, where the rest area is located, and so on.

[0098] The embodiments of the present application can better understand and reconstruct real scenes by identifying the spatial distribution of real objects contained in each image and the distribution of functional areas in the real world, thereby providing a solid foundation for subsequent virtual object generation and interaction.

[0099] Step S403: perform multimodal intent recognition processing on the user's real-time video sequence to obtain the user's real-time intent recognition result.

[0100] The above-mentioned real-time video sequence is obtained by shooting the user in real time. Optionally, the real-time video sequence includes an image sequence of the user and an audio sequence of the user.

[0101] In order to more accurately understand the user's intention, the present application uses at least one of the above-mentioned image sequence and audio sequence to perform multimodal intention recognition processing to obtain the user's real-time intention recognition result. Here, the real-time intention recognition result includes the intention recognition result corresponding to each image in the real-time video sequence.

[0102] The above-mentioned "multimodal intention recognition processing" can be gesture-based intention recognition processing, gaze-based intention recognition processing, body movement-based intention recognition processing, voice-based intention recognition processing, etc., and this application does not make specific limitations.

[0103] It should be noted that the real-time video sequence in this step corresponds one-to-one to the target image sequence mentioned above.

[0104] Step S404: Based on the real-time intention recognition result and the real-time scene understanding model, a real-time scene three-dimensional model that can reflect the interaction mode between the real object and the virtual object is obtained.

[0105] Specifically, the real-time intention recognition result can accurately reflect the user's real intention at each moment, and the real intention refers to the intention of how to make the virtual object interact with the real object. Based on this, based on the real-time intention recognition result and the real-time scene understanding model, a real-time scene 3D model that can reflect the interaction between the real object and the virtual object can be obtained. Here, the real-time scene 3D model includes the scene 3D model corresponding to each image in the target image sequence.

[0106] Optionally, the three-dimensional model of the scene corresponding to each of the above images may be a transformation matrix for interaction between a real object and a virtual object, and the transformation matrix includes position, rotation and scaling information of the real object and the virtual object.

[0107] Optionally, the interaction method includes the following methods: (1) selection, for example, when the user's line of sight or gesture points to a virtual object, the selected object is determined using ray casting technology; (2) movement: by tracking the user's hand movements, the displacement of the hand is directly mapped to the position change of the virtual object and the real object; (3) rotation: detecting the rotation of the user's wrist and applying it to the rotation matrix between the virtual object and the real object; (4) scaling: adjusting the size of the virtual object and / or the real object according to the change in the distance between the hands. Here, the virtual objects and real objects mentioned in the interaction method refer to the virtual objects and real objects presented by AR technology, that is, the virtual objects and real objects presented on the display object.

[0108] Step S405: Perform AR rendering according to the real-time scene 3D model and the target image sequence to obtain an AR rendering result.

[0109] Optionally, the embodiment of the present application can perform AR rendering based on the real-time scene 3D model and the target image sequence based on the bullet physics engine to obtain an AR rendering result. Wherein, the bullet physics engine makes the real objects and virtual objects present physical properties in the AR rendering result under the action of physical parameters.

[0110] That is, the embodiment of the present application can perform AR rendering based on the real-time scene three-dimensional model and the target image sequence. In order to make the interaction process between the real object and the virtual object finally presented on the real object more natural and realistic, the present application can adopt the bullet physics engine and input relevant physical parameters into the bullet physics engine. The bullet physics engine can make the real object and the virtual object present physical properties in the AR rendering result under the action of the physical parameters.

[0111] Optionally, the physical parameters include: gravity parameters, object material parameters, collision parameters, friction parameters and elastic parameters. Of course, the physical parameters in actual applications may be more or less than the above parameters, and this application does not specifically limit them.

[0112] Optionally, the above physical properties include: (1) gravity properties, that is, all virtual objects and real objects are affected by gravity, so that they can fall or stack naturally; (2) collision properties, such as using bullet's discrete collision detection algorithm to accurately calculate the collision between virtual objects and real objects, between virtual objects and other virtual objects, and between real objects and other real objects; (3) friction and elasticity properties: set appropriate friction coefficients and elasticity coefficients for virtual objects and real objects of different materials so that they exhibit real physical properties when colliding and sliding. In particular, the real-time scene understanding model obtained in the previous steps can be used to generate simplified collision bodies (such as convex hulls or simplified polygons) for real objects, so that virtual objects can interact physically with real objects; (4) interactive feedback properties: in order to enhance the user experience, this application can provide multiple forms of feedback, including: visual feedback, auditory feedback and tactile feedback. Among them, visual feedback is to update the rendering position and status of virtual objects and real objects in real time according to the results of physical simulation. For example, when a virtual object collides with a real object, an appropriate deformation or fragmentation effect will be displayed; auditory feedback is to use programmatic audio generation technology to synthesize corresponding sound effects in real time according to physical events (such as collision and friction). Objects of different materials will produce different sounds, which increases the realism of the interaction; tactile feedback is that if the user wears a force feedback device (such as tactile gloves), the corresponding tactile signals can be generated according to the weight, texture and collision of virtual objects and real objects, further enhancing the sense of immersion.

[0113] It should be noted that the above physical properties are only examples and are not intended to limit the present application.

[0114] This embodiment can output the AR rendering result to the display screen of an AR device (such as AR glasses) or a mobile device for user viewing.

[0115] The present application provides a real-time object interaction method based on augmented reality, which can generate a real-time scene understanding model based on a three-dimensional model of a real object in a target image sequence, and the real-time scene understanding model can characterize the spatial distribution of the real object at each shooting moment of the target image sequence and / or the distribution of functional areas in the real world. Therefore, when the spatial relationship of the real objects in the real world photographed changes, it can be reflected in real time on the real-time scene understanding model, and then the real-time scene understanding model is combined with the real-time intention recognition result of the user, so that the behavior of the virtual object can change with the change of the real object, and the interaction between the virtual object and the real object is more realistic. The real-time intention recognition result of the user is obtained based on multimodal consciousness recognition processing. Through multimodal consciousness recognition, the identified real-time intention of the user can be made more accurate, and then the interaction mode between the virtual object and the real object is more consistent with the user's intention, which improves the user experience.

[0116] In addition, the bullet physics engine used in this application requires very few computing resources, which makes the bullet physics engine more compatible with the mobile terminal where the AR device is located, improves the realism and communication fluency of the simulated physical effects, and provides a better user experience.

[0117] In some embodiments of the present application, the process of step S401 “obtaining a three-dimensional model of a real object in a target image sequence” is introduced.

[0118] In order to obtain a more accurate three-dimensional model of a real object, the present application first performs semantic segmentation processing on each image in the target image sequence to obtain a label image corresponding to each image, where each pixel in the label image is annotated with a semantic category label.

[0119] Optionally, the present application may use a semantic segmentation model to perform semantic segmentation on each image in the target image sequence, for example, the DeepLab v3+ semantic segmentation model. DeepLab v3+ is a semantic segmentation model based on an encoder-decoder architecture. The encoder uses a ResNet backbone network with dilated convolutions, which can extract multi-level feature information at different scales. The decoder restores the spatial resolution of the feature map through spatial pyramid pooling and upsampling operations, and generates pixel-level segmentation results.

[0120] Optionally, in order to improve the accuracy of semantic segmentation, the present application improves the semantic segmentation model. Taking the improvement of DeepLab v3+ as an example, the present application introduces a multi-scale feature fusion mechanism in the encoder part, and fuses the shallow high-resolution features with the deep low-resolution features through the feature pyramid network (FPN) to obtain richer multi-scale contextual information; and, a lightweight attention module is designed in the decoder part, which highlights the salient areas and suppresses background interference by adaptively weighting feature maps of different scales. That is, the improved semantic segmentation model in the present application fuses the shallow high-resolution features with the deep low-resolution features in the encoding stage to obtain the fused features as the input of the decoding stage, and performs weighted processing on the decoded feature map in the decoding stage to obtain the weighted feature map as the final output of the decoding stage.

[0121] In the original DeepLab v3+, shallow high-resolution features and deep low-resolution features come from different stages of the encoder (ResNet backbone network), where shallow high-resolution features usually come from the early stages of ResNet (such as Conv1, Layer1, etc.), which retain more spatial detail information, and deep low-resolution features usually come from the later stages of ResNet (such as Layer3, Layer4, etc.), which contain more semantic information but have lower spatial resolution. In this application, a multi-scale feature fusion mechanism (such as FPN) is used in the encoding stage to fuse shallow high-resolution features and deep low-resolution features, which can obtain richer multi-scale context information.

[0122] In the original DeepLab v3+, the original output of the decoding stage is a feature map containing feature information of different scales. This application uses the feature map as input through the attention module, and generates an attention weight map of the same size as the original feature map through a series of convolution and pooling operations. Each element in the attention weight map represents the importance of the feature at the corresponding position in the original feature map. The larger the value, the greater the contribution of the feature at that position to the segmentation result. The attention weight map is multiplied element by element with the feature map of the original output to obtain the weighted feature map as the final output of the decoding stage, so as to generate pixel-level segmentation results.

[0123] Optionally, after improving the semantic segmentation model according to the above process, the present application may also use cross entropy loss and boundary loss (Boundary Loss) to jointly train the improved semantic segmentation model to explicitly supervise the prediction of object boundary pixels to make the segmentation result more refined and smooth.

[0124] After obtaining the label image corresponding to each image in the above text, the present application can perform target detection on the label image corresponding to each image to obtain the recognition result of the real object in each image. Optionally, the recognition result includes a bounding box, a category, and a confidence score of the category.

[0125] Optionally, the present application may use a target detection model to perform target detection on the label image corresponding to each image. The target detection model may be, for example, a Faster R-CNN model. The Faster R-CNN model includes a region proposal network (RPN) and a target classification network. The RPN first generates candidate regions that may contain objects, and then the target classification network classifies and regresses bounding boxes on these candidate regions.

[0126] Optionally, the target detection model may also be a YOLOv5 model. YOLOv5 is a single-stage target detection network that extracts image features through a convolutional neural network and directly regresses the bounding box and category probability of the object on the feature map.

[0127] The present application can improve the target detection model to improve the accuracy of the target detection results. Optionally, the improvements include: introducing an attention mechanism, adding a spatial attention module (Spatial Attention Module) and a channel attention module (Channel Attention Module) at different stages of the backbone network and the neck network, so that the network can adaptively focus on the salient areas and important feature channels of the object, and improve the detection performance. Specifically, the spatial attention module and the channel attention module are added to the Chinese focus layer (Focus), the Chinese cross-stage local layer (CSP) and the feature pyramid network layer (FPN), so that the improved target detection model performs spatial attention weighting and channel attention weighting on the feature maps output by the Chinese focus layer, the Chinese cross-stage local layer and the feature pyramid network layer, respectively, and uses the fusion value of the spatial attention weighted result and the channel attention weighted result as the input of the next layer.

[0128] Taking the Focus layer as an example, assuming that the Focus layer outputs feature map a, the processing process includes: obtaining weight a based on feature map a through the spatial attention module, performing weighted processing on feature map a based on weight a, and obtaining feature map b; obtaining weight b based on feature map a through the channel attention module, performing weighted processing on feature map a based on weight b, and obtaining feature map c; then, feature map b and feature map c are fused to obtain feature map d as the input of the next layer.

[0129] Optionally, the present application may also use a self-supervised pre-training strategy in the training phase of the improved target detection model, by pre-training the backbone network and the neck network on the large-scale image dataset ImageNet to learn a general image feature representation, thereby accelerating the convergence of the model and improving the detection accuracy.

[0130] Furthermore, the embodiments of the present application can process the target image sequence to obtain the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real objects in the target image sequence, and then obtain the initial model of the real objects in each image based on the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real objects in the target image sequence.

[0131] Here, the pose data when the camera captures the target image sequence includes the pose data when the camera captures each image in the target image sequence, and the sparse point cloud information of the real object in the target image sequence includes the sparse point cloud information of the real object contained in each image in the target image sequence.

[0132] Optionally, this embodiment can use the ORB-SLAM2 algorithm to process the target image sequence to obtain the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real object in the target image sequence. Here, the ORB-SLAM2 algorithm is an algorithm that can be used for real-time positioning and mapping. It can use the target image sequence to extract ORB feature points, and estimate the movement of the camera through feature point matching to obtain the pose data when the camera shoots the target image sequence. In addition, the ORB-SLAM2 algorithm can also construct a sparse three-dimensional point cloud map corresponding to the real object, and the sparse three-dimensional point cloud map is the sparse point cloud information in this embodiment.

[0133] Optionally, after obtaining the pose data when the camera takes the target image sequence and the sparse point cloud information of the real object in the target image sequence, the present application can use a monocular depth estimation network such as MonoDepth2 to predict the depth map corresponding to each image in the target image sequence, and then use the truncated signed distance function (TSDF) method to fuse the pose data when the camera takes each image in the target image sequence, the sparse point cloud information of the real object contained in each image in the target image sequence, and the depth map corresponding to each image in the target image sequence to generate a dense three-dimensional point cloud of the real object contained in each image in the target image sequence, and finally use the Poisson surface reconstruction algorithm to convert the dense three-dimensional point cloud into a triangular mesh model to obtain the initial model of the real object in each image. Here, the TSDF method can effectively deal with the noise and occlusion problems in the depth map.

[0134] Optionally, the present application can optimize and update the above-mentioned depth map and pose data by jointly minimizing the reprojection error and the depth consistency error to make the initial model more accurate and complete. Here, the reprojection error is used to measure the difference between the three-dimensional coordinates of the feature points and the camera pose estimated by ORB-SLAM2 and the actually observed pixel coordinates of the feature points; the depth consistency error is used to measure the difference between the depth map predicted by the monocular depth estimation network and the depth of the feature points estimated by ORB-SLAM2.

[0135] Optionally, the present application may also introduce a learning-based loop detection and relocalization algorithm, identify visited scenes through discriminative learning, and relocate the camera pose when tracking fails. That is, in the embodiment of the present application, the camera may move during the shooting process, so that the same real object may be photographed multiple times in the camera. When the camera tracking fails when shooting the real object (such as encountering occlusion, motion blur, etc.), the learning-based loop detection and relocalization algorithm can be used to find the image frame of the camera's previous shooting of the real object, and the camera pose corresponding to the image frame where tracking failed can be re-estimated based on the found image frame.

[0136] Finally, the present application can update the initial model of the real object in each image based on the recognition results of the real object in each image obtained above to obtain a three-dimensional model of the real object in each image; thereby, a three-dimensional model of the real object in the target image sequence can be obtained.

[0137] Optionally, the process of updating the initial model in this embodiment can use a region growing algorithm, starting from the center point of each detected real object, gradually expanding the region according to the spatial proximity of the points and the consistency of the normal vector, and finally obtaining a three-dimensional model of each real object.

[0138] In summary, the embodiments of the present application adopt a variety of improved technologies to make the three-dimensional model of the real object finally modeled more accurate.

[0139] In other embodiments of the present application, the processes of "step S402, generating a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence" and "step S404, obtaining a real-time scene three-dimensional model that can reflect the interaction method between the real object and the virtual object based on the real-time intent recognition result and the real-time scene understanding model" are introduced.

[0140] The embodiment of the present application can first construct a target scene graph corresponding to each image based on the three-dimensional model of the real object in each image.

[0141] Optionally, the process of obtaining the target scene graph in the embodiment of the present application can be obtained by modeling using a graph neural network algorithm, that is, the embodiment of the present application can use a graph neural network algorithm to model based on the three-dimensional model of the real object in each image to obtain the target scene graph corresponding to each image, wherein the target scene graph corresponding to each image has the real objects in the image as nodes, the attribute features of the real objects as node features, and the spatial relationships between the real objects in the image as edges.

[0142] Optionally, the attribute features of the real object can be obtained from the recognition result of the real object obtained by the target detection in the previous text, or after obtaining the recognition result of the real object, based on the category of the recognized real object, external knowledge related to the category is introduced to obtain the attribute features of the real object. Optionally, the attribute features of the real object include: category, position coordinates, size information, orientation, surface material, shape features, etc.

[0143] In order to make the modeled target scene graph contain more prior information, in a preferred implementation method, the scene graph modeled by the graph neural network algorithm can be used as the initial scene graph, that is, the graph neural network algorithm is used to model the three-dimensional model of the real object in each image to obtain the initial scene graph corresponding to each image, and then the semantically associated objects of the real objects in each image are queried from the pre-generated knowledge graph, and the embedding vectors of the semantically associated objects of the real objects in each image are generated, and finally, the initial scene graph corresponding to each image is updated based on the embedding vectors of the semantically associated objects of the real objects in each image to obtain the target scene graph corresponding to each image.

[0144] The above-mentioned pre-generated knowledge graph represents the semantic association between entities in the form of triples (such as <subject, relationship, object>). Therefore, when it is necessary to query the semantically associated object of a real object, the real object can be used as the subject in the triple, and the corresponding relationship and object can be queried. The queried object is the semantically associated object.

[0145] Optionally, this embodiment may use a knowledge representation learning algorithm (such as the TransE algorithm) to embed semantically related objects and queried relationships into a low-dimensional continuous vector space to obtain an embedding vector of the semantically related objects.

[0146] It should be noted that the above process of obtaining the initial scene graph using the graph neural network algorithm is only an example. In addition, other methods can be used to model based on the three-dimensional model of the real object in each image to obtain the initial scene graph corresponding to each image. For example, the initial scene graph is obtained by using a rule-based modeling method, or by using a semantic analysis method, etc. This application does not make specific limitations.

[0147] In this embodiment, the node features in the initial scene graph are the attribute features of the real object in the image, so the embedding vector of the semantically associated object of the real object can be fused with the attribute features of the real object to obtain the fused features of the real object as the features of the node represented by the real image in the target scene graph. If a real object has multiple semantically associated objects, the embedding vectors of the multiple semantically associated objects can be fused with the attribute features of the real object.

[0148] For each real object in each image, feature fusion is performed according to the above process, and the scene graph after feature fusion is used as the target scene graph corresponding to each image.

[0149] Furthermore, the present application can generate a real-time scene understanding model based on the target scene graph corresponding to each image in the target image sequence.

[0150] Optionally, for each image in the target image sequence, a message passing algorithm can be used to obtain a scene global representation vector corresponding to the image based on the target scene graph corresponding to the image, and the scene global representation vector corresponding to the image is used as a scene understanding model corresponding to the image.

[0151] The following introduces the process of using the message passing algorithm to obtain the scene global representation vector corresponding to the image according to the target scene graph corresponding to the image.

[0152] In the first step, for each node in the target scene graph corresponding to the image, based on the adjacency matrix or adjacency list of the graph, the nodes directly connected to the node are obtained as the neighbor nodes of the node, and the fusion features of the neighbor nodes of the node are obtained.

[0153] In the second step, for each node in the target scene graph corresponding to the image, the fusion features of all neighboring nodes of the node are fused according to the dimension to obtain the neighbor features of the node. Optionally, the fusion method of this step can be implemented by an aggregation function, and optionally, the aggregation function can be a sum function or an average function.

[0154] In the third step, for each node in the target scene graph corresponding to the image, the fusion feature of the node is combined with the neighbor feature of the node (e.g., concatenated, added, etc.) to obtain the final feature of the node. Optionally, the combination method of this step can be implemented by an update function, and optionally, the update function can be a function represented by a pre-trained feedforward neural network.

[0155] In the fourth step, for each node in the target scene graph corresponding to the image, the final feature of the node is used as the fusion feature of the node, and the first step is executed again until the total number of returns reaches the threshold. This step is repeated multiple times, so that the node can capture a wider range of context information.

[0156] In the fifth step, a global pooling operation is applied to integrate the final features of all nodes in the target scene graph corresponding to the image obtained in the fourth step into a global representation vector of the scene corresponding to the image.

[0157] After obtaining the scene global representation vector corresponding to each image in the target image sequence as the scene understanding model corresponding to each image in step S402, the process of step S404 "obtaining a real-time scene three-dimensional model that can reflect the interaction method between real objects and virtual objects based on the real-time intention recognition result and the real-time scene understanding model" may include: splicing the intention recognition result corresponding to each image in the real-time video sequence with the scene global representation vector corresponding to the corresponding image in the target image sequence to obtain the real-time scene three-dimensional model.

[0158] The embodiments of the present application comprehensively consider the geometric structure of the real scene as well as the rich semantic information and contextual information, so that a complete three-dimensional model of the real world can be fully understood and reconstructed, allowing virtual objects to interact with real objects more intelligently.

[0159] In some other embodiments of the present application, “Step S403, performing multimodal intent recognition processing on the user's real-time video sequence to obtain a real-time intent recognition result of the user” is explained.

[0160] Taking "multimodal intent recognition processing" including gesture-based intent recognition processing, gaze-based intent recognition processing and voice-based intent recognition processing as an example, optionally, the process of obtaining real-time intent recognition results in this embodiment may include the following steps.

[0161] Firstly, the key points of the user's hands in the real-time video sequence are identified to obtain the gesture recognition result of the user in the real-time video sequence.

[0162] Optionally, a hand key point detection algorithm based on deep learning may be used to identify the hand key points of the user in the real-time video sequence to obtain a gesture recognition result of the user in the real-time video sequence.

[0163] Optionally, the deep learning-based hand key point detection algorithm may be a MediaPipe Hands model, which can detect 21 hand key points in real time, including fingertips, knuckles, and wrists. Based on the detected 21 hand key points, the user gestures can be accurately understood, and further the user intentions reflected by the user gestures can be understood.

[0164] It should be noted that the gesture recognition process shown in this article is only an example. In addition to the hand key point recognition method, other methods can also be used for gesture recognition, such as using a 3D-CNN (three-dimensional convolutional neural network) model to perform spatiotemporal modeling of gestures, capturing the features of gestures in spatial and temporal dimensions, using a temporal pooling operation to aggregate the features of each captured image into a fixed-length feature vector, and then inputting the aggregated feature vector into a fully connected layer for gesture classification to obtain the gesture category and corresponding confidence output by the model. Optionally, the 3D-CNN model can also be improved, such as introducing an attention gate mechanism in the encoding stage or the decoding stage, and dynamically adjusting the weights of different areas of the gesture through the attention gate combined with contextual information to adaptively focus on the salient features of the gesture and improve the accuracy and robustness of gesture recognition.

[0165] Next, voice recognition is performed on the audio data of the user in the real-time video sequence to obtain a voice recognition result of the user in the real-time video sequence.

[0166] Optionally, a preset speech recognition model may be used to perform speech recognition on the audio data of the user in the real-time video sequence to obtain a speech recognition result of the user in the real-time video sequence.

[0167] Optionally, the above-mentioned preset speech recognition model can be a Transformer-based end-to-end speech recognition and semantic understanding model, the encoder of which receives the Mel-spectrogram features of the audio data for encoding, and the encoding result captures the temporal dependency of the speech through the self-attention mechanism. The features obtained by the self-attention mechanism are sent to the decoder, and the decoder generates a corresponding text representation of the input, and then uses a pre-trained language model (such as BERT) to extract features from the generated text representation, and inputs the extracted text features into the intent classifier and slot filler for processing to obtain a speech recognition result, which can be used to predict the user's intent and semantic key information.

[0168] Optionally, for the preset speech recognition model, knowledge distillation technology can also be used to migrate the knowledge of the large speech understanding model to the lightweight preset speech recognition model of this application to facilitate real-time voice interaction.

[0169] Afterwards, the eye movement and head posture of the user in the real-time video sequence are tracked and recognized to obtain the gaze tracking and recognition result of the user in the real-time video sequence.

[0170] Optionally, an eye tracker and a preset head posture estimation algorithm may be used to track and identify the eye movement and head posture of the user in the real-time video sequence to obtain a gaze tracking and identification result of the user in the real-time video sequence.

[0171] The above-mentioned eye tracker can use infrared lighting and a camera to capture the user's eye movement. Optionally, the above-mentioned preset head posture estimation algorithm can be a 6-dimensional posture estimation network based on deep learning, such as DOPE (Deep Object PoseEstimation). The two are combined to track and identify the user's eye movement and head posture, and can accurately calculate the position of the user's gaze point in the real world. The position of the gaze point in the real world is the user's line of sight tracking and identification result in the above-mentioned real-time video sequence.

[0172] It should be noted that the method provided in this application for gaze tracking and recognition by combining an eye tracker and a preset head posture estimation algorithm is only an example. In addition, other methods can also be used, such as using a sequence learning model to model and track eye movement trajectories, and applying an attention mechanism to the output of the sequence learning model, dynamically adjusting the weights of different time steps according to contextual information, and weighting the output of the sequence learning model based on the weight. The weighted result is then fused with the user's visual attention distribution estimated using the gaze point and the law of rapid eye movement to obtain the position of the user's gaze point in the real world, which is used as the user's gaze tracking and recognition result in a real-time video sequence.

[0173] Finally, based on the gesture recognition results, voice recognition results and gaze tracking recognition results, the real-time intention recognition results are obtained.

[0174] Optionally, the present application may use a multimodal fusion model to fuse gesture recognition results, speech recognition results, and gaze tracking recognition results, and obtain intent recognition results based on real-time fusion results. The weights used in the fusion process are weights dynamically adjusted according to different scenarios using an attention mechanism. For example, in scenarios that require precise operations, gesture recognition results may receive higher weights; and in cases where voice commands are more applicable, the weights of voice recognition results will increase accordingly.

[0175] Optionally, considering the continuity of user interaction, the present application can also use a long short-term memory network (LSTM) to model the timing information of gesture recognition results, speech recognition results, and gaze tracking recognition results, so as to obtain the user's real-time intention recognition results based on the timing information.

[0176] In summary, the present application can accurately identify the user's true intention through multimodal intention recognition processing, thereby improving the accuracy of subsequent interaction between virtual objects and real objects.

[0177] In a possible implementation, the process of “performing AR rendering according to the real-time scene three-dimensional model and the target image sequence” in step S405 is explained.

[0178] In the first step, the present embodiment can extract the depth information of each image in the target image sequence to obtain a sparse depth map corresponding to each image. Optionally, a SLAM algorithm is used to extract the depth information of each image in the target image sequence to obtain a sparse depth map corresponding to each image.

[0179] Here, the SLAM (Simultaneous Localization and Mapping) algorithm, that is, simultaneous positioning and mapping, is a technology in which a robot estimates its own position in real time in an unknown environment through its own sensors (such as lidar, camera, inertial measurement unit, etc.) and builds an incremental map of the surrounding environment.

[0180] The second step is to perform dense depth prediction on each image in the target image sequence to obtain a dense depth map corresponding to each image. Optionally, a monocular depth estimation model such as MonoDepth2 is used to perform dense depth prediction on each image in the target image sequence to obtain a dense depth map corresponding to each image.

[0181] Here, MonoDepth2 uses a self-supervised learning method, which does not require a large amount of labeled training data and can generate high-quality depth maps.

[0182] The third step is to perform weighted fusion on the sparse depth map and dense depth map corresponding to each image to obtain a fused depth map corresponding to each image.

[0183] Optionally, the sparse depth map and the dense depth map can be fused by weighted averaging, where the weights are dynamically adjusted based on the confidence of the SLAM depth points and the uncertainty of the monocular depth estimation. During fusion, for areas not covered by SLAM, a larger weight can be set for the dense depth map of the monocular depth estimation.

[0184] Optionally, after obtaining the fused depth map of each image, the embodiment of the present application can also apply a bilateral filter to smooth the fused depth map while retaining the sharpness of the depth edge. This processing method can effectively reduce the noise in the fused depth map while retaining the structural information of the real scene.

[0185] The fourth step is to generate a light and shadow rendering image sequence corresponding to the target image sequence based on the fused depth map, the real-time scene 3D model and the target image sequence corresponding to each image in the target image sequence. Optionally, a pre-trained adversarial generator is used to generate a light and shadow rendering image sequence corresponding to the target image sequence based on the fused depth map, the real-time scene 3D model and the target image sequence corresponding to each image in the target image sequence. Here, the light and shadow rendering image sequence refers to an image sequence composed of light and shadow rendering images corresponding to each image in the target image sequence.

[0186] In order to generate highly realistic AR effects, this step uses deep learning-based neural rendering technology, which can better simulate complex lighting effects and material properties, allowing virtual objects to integrate more naturally into the real world.

[0187] Specifically, this step constructs a neural rendering model based on a generative adversarial network (GAN), which includes an adversarial generator and a discriminator.

[0188] The generator adopts the U-Net architecture, and inputs each image in the target image sequence, the scene 3D model corresponding to the image, the fused depth map corresponding to the image, and the scene illumination estimation information corresponding to the image (optionally, the ambient illumination detection algorithm can be used to extract it from the image) into the generator, and obtains the light and shadow rendering image output by the generator that synthesizes the virtual object and the real object. The light and shadow rendering image output by the generator can achieve seamless fusion of virtual objects and real objects; the discriminator adopts the PatchGAN (Markov discriminator) structure, and its input is the light and shadow rendering image output by the generator or the image in the target image sequence. The goal of the discriminator is to distinguish whether the input image is real or synthetic. Through adversarial training, the generator gradually learns to produce more realistic rendering results, so that the virtual object is highly coordinated with the real environment in terms of lighting, shadows, reflections, etc.

[0189] The fifth step is to perform virtual-real fusion processing based on the light and shadow rendering image sequence and the fused depth map corresponding to each image to obtain a first augmented reality rendering image sequence.

[0190] Optionally, the virtual-real fusion processing includes virtual-real occlusion processing, smoothing processing of virtual object edges and semi-transparent areas, and feathering processing of virtual object edges.

[0191] Optionally, a depth buffer algorithm (Z-buffer) can be used to determine the visible pixels and invisible pixels in the fused depth map corresponding to each image, and then the light and shadow rendering image corresponding to the image is processed for virtual and real occlusion based on the visible pixels and invisible pixels to obtain the first augmented reality rendering image corresponding to the image; the first augmented reality rendering images corresponding to each image in the target image sequence form a first augmented reality rendering image sequence. Here, the virtual objects and real objects in the first augmented reality rendering image sequence have an occlusion relationship consistent with the actual scene.

[0192] Optionally, after obtaining the first augmented reality rendering image sequence, an Alpha blending algorithm may be used to smooth edge areas or semi-transparent areas of virtual objects in the first augmented reality rendering image sequence, wherein the blending weight used by the Alpha blending algorithm may be dynamically calculated based on the Alpha channel value of the virtual object.

[0193] Optionally, in order to further improve the naturalness of the virtual-real boundary, the present application may also apply an adaptive feathering algorithm to the edge of the virtual object for feathering processing, wherein the feathering degree is dynamically adjusted according to the depth difference and edge complexity, so that the virtual object can be more naturally integrated into the real scene.

[0194] In a sixth step, a preset post-processing is performed on the first augmented reality rendering image sequence to obtain a second augmented reality rendering image sequence with a stronger sense of reality, wherein the preset post-processing includes at least one of the following processing: super-resolution reconstruction, color correction processing, anti-aliasing processing, global illumination processing, blur adjustment processing, and preset lens information adding processing.

[0195] Taking the preset post-processing including super-resolution reconstruction, color correction processing, anti-aliasing processing, global illumination processing, blur adjustment processing and preset lens information adding processing as an example, an optional processing process may include:

[0196] Each image in the first augmented reality rendering image sequence is reconstructed with high resolution to obtain a third augmented reality rendering image sequence.

[0197] Optionally, the ESRGAN (Enhanced Super-Resolution GenerativeAdversarial Networks) model can be used to perform high-resolution reconstruction on each first augmented reality rendering image in the first augmented reality rendering image sequence. Here, the training data of the ESRGAN model includes a large number of training first augmented reality rendering images, covering various lighting conditions and scene types, so that the trained ESRGAN model can enhance the clarity and details of virtual objects while maintaining the details of the real scene. The super-resolution reconstruction of this embodiment has the following advantages: improving the texture details of virtual objects; enhancing the clarity of the virtual-real boundary; and improving the visual quality of the overall image.

[0198] In order to ensure that the virtual object and the real object can maintain visual consistency under different lighting environments, this embodiment can perform color correction processing on each image in the third augmented reality rendering image sequence to obtain a fourth augmented reality rendering image sequence.

[0199] Optionally, an adaptive color correction and tone mapping algorithm can be used for color correction processing. The adaptive color correction and tone mapping algorithm first estimates the color temperature and overall brightness level of the real environment in the real scene by analyzing the color distribution and brightness histogram of the target image sequence, and then dynamically adjusts the rendering parameters of the virtual object according to these parameters, including adjusting the color temperature of the virtual light source to match the ambient light, and applying color gain and offset to match the overall hue of the scene.

[0200] This embodiment uses an adaptive color correction and tone mapping algorithm to ensure that virtual objects and real objects maintain appropriate contrast and details in high dynamic range scenes.

[0201] In order to avoid flickering or instability between frames, this embodiment may further perform anti-aliasing processing on each image in the fourth augmented reality rendering image sequence to obtain a fifth augmented reality rendering image sequence.

[0202] Optionally, a multi-sampling anti-aliasing (MSAA) algorithm may be used for anti-aliasing processing, and the MSAA algorithm has the following processing steps: using temporal anti-aliasing (TAA) technology to smooth the edges of virtual objects in combination with information of current images and historical images; interpolating the motion of virtual objects to ensure smooth animation effects even at low frame rates; and progressively rendering the fourth augmented reality rendering image sequence, computing loads between multiple images, and balancing image quality and real-time performance.

[0203] This embodiment may also perform global illumination processing, blur adjustment processing, and preset lens information adding processing on each image in the fifth augmented reality rendering image sequence in sequence to obtain a second augmented reality rendering image sequence.

[0204] Optionally, a global illumination approximation algorithm, such as SSAO (screen space ambient occlusion) algorithm, may be used for global illumination processing. The above blur adjustment processing refers to dynamically adjusting the blur degree of virtual objects and real objects according to the user's focus based on the physical depth of field effect.

[0205] Optionally, the preset lens information adding process may be a subtle lens effect adding process, such as a chromatic aberration and lens glare adding process, and adding this information may help improve the realism of the virtual object.

[0206] Through the above series of complex rendering and image processing technologies, the present application can generate highly realistic and visually seamless AR rendering results, in which virtual objects are not only precisely aligned with real objects in geometry, but also highly consistent in lighting, materials and overall visual effects. In order to improve rendering efficiency, this embodiment can use GPU (Graphics Processing Unit) parallel computing, and can also use shader programs to ensure that while maintaining high visual quality, a frame rate of at least 60fps can be achieved, providing users with a smooth AR experience.

[0207] A real-time object interaction method based on augmented reality provided in an embodiment of the present application is introduced above. A device for executing the real-time object interaction method based on augmented reality is introduced below.

[0208] See also Figure 5 , Figure 5 A schematic diagram of a real-time object interaction device based on augmented reality provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device may include:

[0209] A model acquisition module 501 is used to acquire a three-dimensional model of a real object in a target image sequence, wherein the target image sequence is obtained by continuously photographing the real world;

[0210] A first model generation module 502 is used to generate a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence, wherein the real-time scene understanding model includes a scene understanding model corresponding to each image in the target image sequence, and the scene understanding model corresponding to each image can characterize the spatial relationship between the real objects in the image and the distribution of functional areas in the real scene corresponding to the image;

[0211] The intention recognition module 503 is used to perform multimodal intention recognition processing on the user's real-time video sequence to obtain the user's real-time intention recognition result;

[0212] A second model generation module 504 is used to obtain a real-time scene three-dimensional model that can reflect the interaction mode between the real object and the virtual object based on the real-time intention recognition result and the real-time scene understanding model;

[0213] The AR rendering module 505 is used to perform AR rendering according to the real-time scene 3D model and the target image sequence to obtain an AR rendering result.

[0214] In a possible implementation, the process of the model acquisition module acquiring the three-dimensional model of the real object in the target image sequence may include:

[0215] An improved semantic segmentation model is used to perform semantic segmentation processing on each image in the target image sequence to obtain a label image corresponding to each image, wherein each pixel in the label image is annotated with a semantic category label. The improved semantic segmentation model fuses shallow high-resolution features with deep low-resolution features in the encoding stage to obtain fused features as input of the decoding stage. In the decoding stage, the decoded feature map is weighted to obtain a weighted feature map as the final output of the decoding stage.

[0216] An improved target detection model is used to perform target detection on the label image corresponding to each image, and the recognition result of the real object in each image is obtained. The improved target detection model performs spatial attention weighting and channel attention weighting on the feature maps output by the Chinese focus layer, the Chinese cross-stage local layer and the feature pyramid network layer, respectively, and uses the fusion value of the spatial attention weighting result and the channel attention weighting result as the input of the next layer;

[0217] Processing the target image sequence to obtain the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real object in the target image sequence;

[0218] Based on the pose data of the camera when shooting the target image sequence and the sparse point cloud information of the real object in the target image sequence, an initial model of the real object in each image is obtained;

[0219] Based on the recognition result of the real object in each image, the initial model of the real object in each image is updated to obtain the three-dimensional model of the real object in each image; so as to obtain the three-dimensional model of the real object in the target image sequence.

[0220] In a possible implementation, the process of generating a real-time scene understanding model by the first model generation module based on the three-dimensional model of the real object in the target image sequence may include:

[0221] Based on the three-dimensional model of the real object in each image, a target scene graph corresponding to each image is constructed, wherein the target scene graph corresponding to each image has the real object in the image as a node, the attribute features of the real object as a node feature, and the spatial relationship between the real objects in the image as an edge;

[0222] A real-time scene understanding model is generated based on the target scene graphs corresponding to each image in the target image sequence.

[0223] In a possible implementation, the process of the first model generation module constructing a target scene graph corresponding to each image based on the three-dimensional model of the real object in each image may include:

[0224] Modeling is performed based on the three-dimensional model of the real object in each image to obtain the initial scene graph corresponding to each image;

[0225] Query the semantically related objects of the real objects in each image from the pre-generated knowledge graph, and generate the embedding vectors of the semantically related objects of the real objects in each image;

[0226] The initial scene graph corresponding to each image is updated based on the embedding vector of the semantically associated object of the real object in each image to obtain the target scene graph corresponding to each image.

[0227] In a possible implementation, the intention recognition module performs multimodal intention recognition processing on the user's real-time video sequence, and the process of obtaining the user's real-time intention recognition result may include:

[0228] Identify the key points of the user's hand in the real-time video sequence to obtain the gesture recognition result of the user in the real-time video sequence;

[0229] Performing speech recognition on the audio data of the user in the real-time video sequence to obtain a speech recognition result of the user in the real-time video sequence;

[0230] Track and identify the user's eye movement and head posture in the real-time video sequence to obtain the user's sight tracking and identification results in the real-time video sequence;

[0231] Based on the gesture recognition results, voice recognition results and gaze tracking recognition results, real-time intention recognition results are obtained.

[0232] In a possible implementation, the process of performing AR rendering by the AR rendering module according to the real-time scene 3D model and the target image sequence may include:

[0233] Extract the depth information of each image in the target image sequence to obtain a sparse depth map corresponding to each image;

[0234] Perform dense depth prediction on each image in the target image sequence to obtain a dense depth map corresponding to each image;

[0235] Perform weighted fusion on the sparse depth map and the dense depth map corresponding to each image to obtain a fused depth map corresponding to each image;

[0236] Based on the fused depth map, the real-time scene three-dimensional model and the target image sequence respectively corresponding to each image in the target image sequence, a light and shadow rendering image sequence corresponding to the target image sequence is generated;

[0237] Performing virtual-real fusion processing based on the light and shadow rendering image sequence and the fused depth map corresponding to each image to obtain a first augmented reality rendering image sequence, wherein the virtual-real fusion processing includes virtual-real occlusion processing, smoothing processing of virtual object edges and semi-transparent areas, and feathering processing of virtual object edges;

[0238] The first augmented reality rendering image sequence is subjected to preset post-processing to obtain a second augmented reality rendering image sequence with a stronger sense of reality, wherein the preset post-processing includes at least one of the following processing: super-resolution reconstruction, color correction processing, anti-aliasing processing, global illumination processing, blur adjustment processing, and preset lens information adding processing.

[0239] The present application also provides an electronic device in an embodiment. Figure 6As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0240] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0241] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0242] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the real-time object interaction methods based on augmented reality provided in the embodiments of the present application.

[0243] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the real-time object interaction methods based on augmented reality provided in the embodiment of the present application.

[0244] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0245] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0246] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0247] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A real-time object interaction method based on augmented reality, characterized in that: include: Acquire a three-dimensional model of a real object in a target image sequence, wherein the target image sequence is obtained by continuously photographing the real world; Generate a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence, wherein the real-time scene understanding model includes a scene understanding model corresponding to each image in the target image sequence, and the scene understanding model corresponding to each image can characterize the spatial relationship between the real objects in the image and the distribution of functional areas in the real scene corresponding to the image; Performing multimodal intention recognition processing on the user's real-time video sequence to obtain a real-time intention recognition result of the user; Based on the real-time intention recognition result and the real-time scene understanding model, a real-time scene three-dimensional model capable of reflecting the interaction mode between the real object and the virtual object is obtained; AR rendering is performed according to the real-time scene three-dimensional model and the target image sequence to obtain an AR rendering result.

2. The real-time object interaction method based on augmented reality according to claim 1, characterized in that: The step of obtaining a three-dimensional model of a real object in a target image sequence comprises: Using an improved semantic segmentation model to perform semantic segmentation processing on each image in the target image sequence to obtain a label image corresponding to each image, wherein each pixel in the label image is annotated with a semantic category label, and the improved semantic segmentation model fuses shallow high-resolution features with deep low-resolution features in the encoding stage to obtain fused features as input of the decoding stage, and performs weighted processing on the decoded feature map in the decoding stage to obtain a weighted feature map as the final output of the decoding stage; An improved target detection model is used to perform target detection on the label image corresponding to each image to obtain the recognition result of the real object in each image, wherein the improved target detection model performs spatial attention weighting and channel attention weighting on the feature maps output by the Chinese focus layer, the Chinese cross-stage local layer and the feature pyramid network layer, respectively, and uses the fusion value of the spatial attention weighted result and the channel attention weighted result as the input of the next layer; Processing the target image sequence to obtain the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real object in the target image sequence; Based on the pose data when the camera shoots the target image sequence and the sparse point cloud information of the real object in the target image sequence, an initial model of the real object in each image is obtained; Based on the recognition result of the real object in each image, the initial model of the real object in each image is updated to obtain the three-dimensional model of the real object in each image; so as to obtain the three-dimensional model of the real object in the target image sequence.

3. The real-time object interaction method based on augmented reality according to claim 2, characterized in that: The generating a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence comprises: Based on the three-dimensional model of the real object in each image, a target scene graph corresponding to each image is constructed, wherein the target scene graph corresponding to each image has the real object in the image as a node, the attribute features of the real object as a node feature, and the spatial relationship between the real objects in the image as an edge; The real-time scene understanding model is generated based on the target scene graphs corresponding to the respective images in the target image sequence.

4. The real-time object interaction method based on augmented reality according to claim 2, characterized in that: The step of constructing a target scene graph corresponding to each image based on the three-dimensional model of the real object in each image comprises: Modeling is performed based on the three-dimensional model of the real object in each image to obtain an initial scene graph corresponding to each image; Querying semantically associated objects of the real object in each image from a pre-generated knowledge graph, and generating an embedding vector of the semantically associated objects of the real object in each image; The initial scene graph corresponding to each image is updated based on the embedding vector of the semantically associated object of the real object in each image to obtain the target scene graph corresponding to each image.

5. The real-time object interaction method based on augmented reality according to claim 1, characterized in that: The performing multimodal intention recognition processing on the real-time video sequence of the user to obtain the real-time intention recognition result of the user includes: Identify key points of the user's hand in the real-time video sequence to obtain a gesture recognition result of the user in the real-time video sequence; Performing speech recognition on the audio data of the user in the real-time video sequence to obtain a speech recognition result of the user in the real-time video sequence; Tracking and identifying the eye movement and head posture of the user in the real-time video sequence to obtain a sight line tracking and identification result of the user in the real-time video sequence; The real-time intention recognition result is obtained based on the gesture recognition result, the voice recognition result and the sight tracking recognition result.

6. The real-time object interaction method based on augmented reality according to claim 1, characterized in that: The performing AR rendering according to the real-time scene three-dimensional model and the target image sequence includes: Extracting depth information of each image in the target image sequence to obtain a sparse depth map corresponding to each image; Performing dense depth prediction on each image in the target image sequence to obtain a dense depth map corresponding to each image; Performing weighted fusion on the sparse depth map and the dense depth map corresponding to each image to obtain a fused depth map corresponding to each image; Based on the fused depth maps respectively corresponding to the images in the target image sequence, the real-time scene three-dimensional model and the target image sequence, generating a light and shadow rendering image sequence corresponding to the target image sequence; Performing virtual-real fusion processing based on the light and shadow rendering image sequence and the fused depth map corresponding to each image to obtain a first augmented reality rendering image sequence, wherein the virtual-real fusion processing includes virtual-real occlusion processing, smoothing processing of virtual object edges and semi-transparent areas, and feathering processing of virtual object edges; The first augmented reality rendering image sequence is subjected to preset post-processing to obtain a second augmented reality rendering image sequence with a stronger sense of reality, wherein the preset post-processing includes at least one of the following processing: super-resolution reconstruction, color correction processing, anti-aliasing processing, global illumination processing, blur adjustment processing and preset lens information adding processing.

7. A real-time object interaction device based on augmented reality, characterized in that: include: A model acquisition module, used to acquire a three-dimensional model of a real object in a target image sequence, wherein the target image sequence is obtained by continuously photographing the real world; A first model generation module is used to generate a real-time scene understanding model based on the three-dimensional model of the real object in the target image sequence, wherein the real-time scene understanding model includes a scene understanding model corresponding to each image in the target image sequence, and the scene understanding model corresponding to each image can characterize the spatial relationship between the real objects in the image and the distribution of functional areas in the real scene corresponding to the image; An intention recognition module is used to perform multimodal intention recognition processing on a user's real-time video sequence to obtain a real-time intention recognition result of the user; A second model generation module is used to obtain a real-time scene three-dimensional model that can reflect the interaction mode between the real object and the virtual object based on the real-time intention recognition result and the real-time scene understanding model; The AR rendering module is used to perform AR rendering according to the real-time scene three-dimensional model and the target image sequence to obtain an AR rendering result.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the real-time object interaction method based on augmented reality as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the real-time object interaction method based on augmented reality as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the real-time object interaction method based on augmented reality as described in any one of claims 1 to 6.

Images