Video interaction method and device and related equipment

Abstract

The invention discloses a video interaction method and device and related equipment, and belongs to the technical field of videos. The method comprises: receiving first to-be-processed data, the first to-be-processed data comprising a first video call picture and a target object specified by a user in the first video call picture, and the first video call picture being generated in a video call process between a first terminal and a second terminal; processing the first to-be-processed data to obtain first target data, the first target data comprising a first target picture and a first 3D model of a target object, and the first target picture being a picture after the target object in the first video call picture is deleted; and sending the first target data to the first terminal and the second terminal, so that the first terminal and the second terminal update the first video call picture according to the first target picture and the first 3D model. The user can describe the target object more intuitively, and the communication efficiency of both parties of the video call is improved.

Description

Technical Field

[0001] The present application belongs to the field of video technology, and specifically relates to a video interaction method, device and related equipment. Background Art

[0002] Video call technology has brought convenience to people's communication. However, when both parties in a video call need to explain a certain object in the video in detail, the existing method is usually for one user to verbally describe the object that the user is interested in. However, verbal description requires a certain level of language expression ability, otherwise the other user is likely to misunderstand, resulting in the two parties not talking about the same object.

[0003] It can be seen that the existing video interaction method cannot directly interact with the video call content, and there is a problem of low communication efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video interaction method, apparatus and related equipment, which can solve the problem of low communication efficiency of existing video interaction methods.

[0005] In a first aspect, an embodiment of the present application provides a video interaction method, the method comprising:

[0006] Receiving first data to be processed, wherein the first data to be processed includes a first video call screen and a target object specified by a user in the first video call screen, and the first video call screen is generated during a video call between a first terminal and a second terminal;

[0007] Processing the first data to be processed to obtain first target data, wherein the first target data includes a first target screen and a first 3D model of the target object, and the first target screen is a screen after deleting the target object in the first video call screen;

[0008] The first target data is sent to the first terminal and the second terminal, so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model.

[0009] In a second aspect, an embodiment of the present application provides a video interaction device, including:

[0010] A first receiving module, configured to receive first data to be processed, wherein the first data to be processed includes a first video call screen and a target object specified by a user in the first video call screen, wherein the first video call screen is generated during a video call between a first terminal and a second terminal;

[0011] A first processing module is configured to process the first data to be processed to obtain first target data, wherein the first target data includes a first target screen and a first 3D model of the target object, and the first target screen is a screen after deleting the target object in the first video call screen;

[0012] The first sending module is used to send the first target data to the first terminal and the second terminal, so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0016] In an embodiment of the present application, during a video call between a first terminal and a second terminal, a first server receives first data to be processed, the first data to be processed including a first video call screen and a target object specified by a user in the first video call screen; then the first server processes the first data to be processed to obtain first target data including a first target screen and a first 3D model; and then sends the first target data to the first terminal and the second terminal respectively, so that the first terminal and the second terminal can synchronously present the first target screen and the first 3D model of the target object, and the first 3D model is superimposed and displayed on the first target screen. In this way, the user can still conduct normal video communication through the first target screen, and when describing the target object based on the first 3D module, the user can describe the target object more intuitively, reducing the situation where repeated confirmation and communication are required due to unclear description or unclear indication of the target object by other means, thereby improving the communication efficiency of both parties in the video call. It can be applied to various video call scenarios, especially when the target object needs to be accurately described or operated, such as teaching, work meetings, remote technical support, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 One of the flowcharts of a video interaction method provided in an embodiment of the present application;

[0019] Figure 2 A second flowchart of a video interaction method provided in an embodiment of the present application;

[0020] Figure 3 A third flowchart of a video interaction method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of a video interaction device provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] The method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0026] like Figure 1As shown, it is one of the flowcharts of a video interaction method provided in an embodiment of the present application. The execution subject of the video interaction method of this embodiment may be a first server, and specifically includes the following steps:

[0027] Step 101: receiving first data to be processed, wherein the first data to be processed includes a first video call screen and a target object specified by a user in the first video call screen, wherein the first video call screen is generated during a video call between a first terminal and a second terminal;

[0028] The first terminal may be a terminal device used by user A, such as a mobile phone, a tablet computer, a videophone, etc.; the second terminal may be a terminal device used by user B, such as a mobile phone, a tablet computer, a videophone, etc. The first terminal may collect the image of the environment where user A is located, and send it to the second terminal through the first terminal, so as to present the image of the environment where user A is located to user B in the second terminal; similarly, the second terminal may collect the image of the environment where user B is located, and send it to the first terminal through the second terminal, so as to present the image of the environment where user B is located to user A in the first terminal, thereby realizing a real-time video call between user A and user B.

[0029] The first server may be a server corresponding to the first terminal, that is, the background of the first terminal. When user A and user B respectively conduct a video call through the corresponding first terminal and second terminal, when user A wants to explain the target object in the video screen, for example, user A wants to introduce the target object in the video call in detail to user B, if the target object is an object in the environment where user A is located, user A can pick up the target object and bring the target object close to the first terminal, so that user B can clearly view the target object on the second terminal based on the communication connection between the first terminal and the second terminal. However, when the target object is an object in the environment where user B is located, or when user A cannot pick up the target object, user A introduces the target object in detail to user B during the video call, as described below:

[0030] In one example, when the first video call screen is the first video call screen captured by the camera of the second terminal, that is, the target object is an object in the environment where user B is located, the first terminal and the second terminal synchronously display the first video call screen, and user A makes a first input to the target object in the first video call screen on the first terminal. The first terminal generates first data to be processed based on the first input of user A, and sends the first data to be processed to the first server. The first data to be processed includes the first video call screen and location information of the target object in the first video call screen.

[0031] In another example, when the first video call screen is the first video call screen captured by the camera of the first terminal, that is, the target object is an object in the environment where user A is located, the first terminal and the second terminal synchronously display the first video call screen, and user A can also make a first input to the target object in the first video call screen on the first terminal. The first terminal generates first data to be processed based on the first input of user A, and sends the first data to be processed to the first server. The first data to be processed includes the first video call screen and location information of the target object in the first video call screen.

[0032] In the above example, the first data to be processed can be sent directly from the first terminal to the first server. It should be understood that the first data to be processed can also be sent to the first server via other transmission media, so that the first server receives the first data to be processed from the first terminal, which can also achieve the same technical effect, and will not be repeated here.

[0033] The location information of the target object specified by the user in the first video call screen may be the first input of the target object in the first video call screen by the user on the first terminal to determine the location information of the target object in the first video call screen. For details, please refer to the following description:

[0034] In one example, the first terminal and the second terminal synchronously display the first video call screen. When user A specifies the target object in the first video call screen, he can select the target object by long pressing the screen of the first terminal. To ensure that user A is clear about his choice, the first terminal can overlay a conspicuous color in the display area of ​​the target object in the first video call screen as a prompt. In this way, user A completes the first input on the first terminal; the first terminal generates the first data to be processed based on the first input of user A, and sends the first data to be processed to the first server.

[0035] In another example, the first terminal and the second terminal synchronously display the first video call screen. When user A specifies the target object in the first video call screen, the target object can be selected by voice input. To ensure that user A is clear about his choice, the first terminal can overlay a conspicuous color in the display area of ​​the target object in the first video call screen as a prompt. In this way, user A completes the first input on the first terminal; the first terminal generates the first data to be processed based on the first input of user A, and sends the first data to be processed to the first server.

[0036] In another example, the first terminal and the second terminal synchronously display the first video call screen. When user A specifies the target object in the first video call screen, the target object can be selected by taking a screenshot and using image recognition. To ensure that user A is clear about his or her choice, the first terminal can overlay a conspicuous color in the display area of ​​the target object in the first video call screen as a prompt. In this way, user A completes the first input on the first terminal; the first terminal generates the first data to be processed based on the first input of user A, and sends the first data to be processed to the first server.

[0037] It should be understood that, for user B, the above method can also be used to specify the target object in the first video call screen to determine the location information of the target object in the first video call screen.

[0038] The first server receives the first data to be processed sent by the first terminal, and further processes the first data to be processed in subsequent step 102, which can be described as follows:

[0039] Step 102: Process the first data to be processed to obtain first target data, wherein the first target data includes a first target screen and a first 3D model of the target object, and the first target screen is a screen after deleting the target object in the first video call screen;

[0040] The first data to be processed includes the first video call screen and the location information of the target object in the first video call screen. When the first server processes the first data to be processed, it can perform three-dimensional (3D) modeling based on the image information of the target object in the first video call screen to obtain a first 3D model of the target object; and delete the target object in the first video call screen based on the location information of the target object in the first video call screen to obtain the first target screen. The first 3D model can be superimposed and displayed on the first target screen, and user A and user B can still communicate normally via video through the first target screen. When describing the target object based on the first 3D module, the user can see and describe the target object more intuitively, reducing misunderstandings caused by vague descriptions or unclear indication of the target object through other means.

[0041] In some examples, the processing the first to-be-processed data to obtain first target data includes:

[0042] Deleting the target object from the first video call screen according to the position information of the target object in the first video call screen to obtain a first intermediate screen;

[0043] Image restoration is performed on the position in the first intermediate picture where the target object is deleted to obtain the first target picture.

[0044] In this example, the first server deletes the target object in the first video call screen according to the position information of the target object in the first video call screen, that is, performs a cutout process on the position where the target object is located to obtain a first intermediate screen; then, using neural network technology, the first server performs image repair on the position where the target object is deleted in the first intermediate screen, and reconstructs the background of the position to present a real environment. In order to ensure that the target object attracts the user's attention, the first server can also add a special effect to the first target screen, so that the target object alternates between display and hiding, and applies a flashing effect when it is visible. This can more effectively attract the user's attention and help the user quickly locate the target object during the video call.

[0045] Step 103: Send the first target data to the first terminal and the second terminal, so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model.

[0046] In an example, the first server may send the first target data to the first terminal, so that the first terminal updates the first video call screen according to the first target screen and the first 3D model.

[0047] In another example, the first server may send the first target data to the first terminal and the second terminal respectively, so that the first terminal and the second terminal respectively update the displayed first video call screen.

[0048] The updating of the displayed first video call screen includes: updating the displayed first video call screen to the first target screen, and overlaying and displaying the first 3D model of the target object in the first target screen.

[0049] The first server sends the first target data to the first terminal and the second terminal respectively, so that the first terminal and the second terminal can synchronously present the first target screen and the first 3D model of the target object, and the first 3D model is superimposed and displayed on the first target screen. The user can still conduct normal video communication through the first target screen. When describing the target object based on the first 3D module, the user can describe the target object more intuitively, reducing the need for repeated confirmation and communication due to unclear descriptions or unclear indication of the target object through other means, thereby improving the communication efficiency of both parties in the video call. It can be applied to various video call scenarios, especially when it is necessary to accurately describe or operate the target object, such as teaching, work meetings, remote technical support, etc.

[0050] In the embodiment of the present application, during the video call between the first terminal and the second terminal, the first server receives the first data to be processed, the first data to be processed includes the first video call screen and the target object specified by the user in the first video call screen; then the first server processes the first data to be processed to obtain the first target data including the first target screen and the first 3D model; and then sends the first target data to the first terminal and the second terminal respectively, so that the first terminal and the second terminal can synchronously present the first target screen and the first 3D model of the target object, and the first 3D model is superimposed and displayed on the first target screen. In this way, the user can still conduct normal video communication through the first target screen, and when describing the target object based on the first 3D module, the user can describe the target object more intuitively, reducing the situation where repeated confirmation and communication are required due to unclear description or unclear indication of the target object by other means, thereby improving the communication efficiency of both parties in the video call. It can be applied to various video call scenarios, especially when the target object needs to be accurately described or operated, such as teaching, work meetings, remote technical support, etc.

[0051] In some optional embodiments, after sending the first target data to the first terminal and the second terminal so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model, the method further includes:

[0052] In case of receiving a first request from the first terminal or the second terminal, in response to the first request, updating the first 3D model to obtain a second 3D model, wherein the first request is used to request updating at least one of the following parameters of the first 3D model: display position, model posture, and model size;

[0053] The second 3D model is sent to the first terminal and the second terminal, so that the first terminal and the second terminal respectively update the displayed first 3D model.

[0054] In one example, user A uses the first terminal and user B uses the second terminal. When user A and user B make a video call through the corresponding first terminal and second terminal respectively, the first terminal and the second terminal synchronously display the first target screen and the first 3D model of the target object. At this time, when user A wants to further introduce the first 3D model superimposed on the first target screen in a more detailed and comprehensive manner, user A can make a second input to the displayed first 3D model on the first terminal. The first terminal generates a first request based on the second input of user A and sends the first request to the first server. The first request and the user request update at least one parameter of the display position, model posture and model size of the first 3D model. When the first server receives the first request from the first terminal, it responds to the first request and updates the first 3D model to obtain the second 3D model.

[0055] Exemplarily, user A may perform a second input on the displayed first 3D model on the first terminal, as described below:

[0056] The first terminal and the second terminal synchronously display the first target screen and the first 3D model superimposed on the first target screen. When user A wants to further introduce the first 3D model superimposed on the first target screen in a more detailed and comprehensive manner, user A can press / double-click the first 3D module on the first terminal with a finger, and when the first 3D model is selected, adjust the display position of the first 3D model by sliding with three fingers (i.e., moving the first 3D model); slide the model posture of the first 3D model by two fingers (i.e., rotating the first 3D model); reduce the first 3D model by sliding two fingers toward each other, and enlarge the first 3D model by sliding two fingers away from each other (i.e., scaling the first 3D model). In this way, user A completes the second input on the first terminal; the first terminal generates a first request based on the second input of user A, and sends the first request to the first server. When the first server receives the first request of the first terminal, it parses and responds to the first request, updates the first 3D model, and obtains the second 3D model. The first terminal and the second terminal synchronously display the second 3D model in real time, realizing the process of interacting with the three-dimensional model through gestures (such as moving, rotating and scaling, etc.), enriching the display content of the video call process, and further improving the communication efficiency of both parties in the video call.

[0057] In some other optional embodiments, when the first video call screen is a video call screen generated according to a screen captured by a camera of the second terminal during a video call between the first terminal and the second terminal, after sending the first target data to the first terminal and the second terminal so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model, the method further includes:

[0058] receiving second data to be processed, wherein the second data to be processed includes a second video call screen and a posture change amount of a camera of the second terminal, and the second video call screen is a video call screen generated according to a screen captured by the camera of the second terminal after the posture of the camera of the second terminal changes;

[0059] Processing the second data to be processed and the first 3D model to obtain second target data, wherein the second target data includes a second target screen and a third 3D model of the target object, the second target screen is a screen after deleting the target object in the second video call screen, and the third 3D model is a 3D model obtained by updating the posture of the first 3D model based on the posture change amount;

[0060] The second target data is sent to the first terminal and the second terminal, so that the first terminal and the second terminal update the displayed video call screen according to the second target screen and the third 3D model.

[0061] Exemplarily, the first server may send the second target data to the first terminal and the second terminal respectively, so that the first terminal and the second terminal respectively update the displayed video call screen, wherein the updating of the displayed video call screen includes: updating the first target screen to the second target screen, and updating the first 3D model to the third 3D model.

[0062] In this embodiment, Figure 2 As shown, an exemplary description is given by taking the first video call screen as a video call screen generated according to the camera capture screen of the second terminal during the video call between the first terminal and the second terminal. It should be understood that when the first video call screen is a video call screen generated according to the camera capture screen of the first terminal during the video call between the first terminal and the second terminal, the same technical effect can also be achieved.

[0063] The first terminal may be a terminal device used by user A; the second terminal may be a terminal device used by user B. The camera of the first terminal may capture the image of the environment where user A is located, and send it to the second terminal through the first terminal, so as to present the image of the environment where user A is located to user B on the screen of the second terminal; similarly, the camera of the second terminal may capture the image of the environment where user B is located, and send it to the first terminal through the second terminal, so as to present the image of the environment where user B is located to user A on the screen of the first terminal, thereby realizing a real-time video call between user A and user B. User A makes a first input to the target object in the first video call screen on the first terminal, and the first terminal generates first data to be processed based on the first input of user A, and sends the first data to be processed to the first server; the first server processes the first data to be processed to obtain the first target data; then, the first server sends the first target data to the first terminal, so as to display the first target screen in the first terminal, and superimpose the first 3D model of the target object in the first target screen; at the same time, the first target data is synchronized to the second terminal through the first terminal, so as to display the first target screen in the second terminal, and superimpose the first 3D model of the target object in the first target screen.

[0064] Further, such as Figure 3 As shown, the second server (i.e., the server corresponding to the second terminal) records the posture of the camera of the second terminal in real time. When the first server obtains that the posture of the camera of the second terminal has changed, for example, user B moves the second terminal, and the posture of the camera of the second terminal changes accordingly, the first server receives the second data to be processed sent by the second terminal, wherein the second data to be processed includes the second video call picture and the change in posture of the camera of the second terminal. The second video call picture is the picture captured by the camera of the second terminal after the posture of the camera of the second terminal has changed (i.e., the picture of the environment where user B is located), and the second terminal synchronizes the captured picture to the first terminal.

[0065] The first terminal sends the picture captured by the camera of the second terminal to the first server (i.e., the second video call picture, that is, the picture of the environment where user B is located captured by the camera of the second terminal after the posture of the camera of the second terminal changes). Then, the first server performs picture update processing on the second video call picture to obtain a second target picture, and performs model update processing on the first 3D model to obtain a second 3D model. The second target picture is the picture after deleting the target object in the second video call picture, and the third 3D model is a 3D model obtained by updating the posture of the first 3D model based on the posture change amount.

[0066] The first server sends the second target data to the first terminal and the second terminal respectively, so that the first terminal and the second terminal can update the presented first target screen to the second target screen, and update the presented first 3D model to the second 3D model, and the second 3D model is superimposed and displayed on the second target screen, so as to update the displayed video call screen. The spatial position of the target object is tracked in real time by using Simultaneous Localization and Mapping (SLAM) technology, and corresponding adjustments are made according to the position and posture of the mobile phone, thereby improving the communication efficiency of both parties in the video call. It can be more widely applied to various video call scenarios, especially when the target object needs to be accurately described or operated, such as teaching, work meetings, remote technical support, etc.

[0067] Among them, in the case that the first video call screen is the first video call screen captured by the camera of the second terminal, that is, the target object is an object in the environment where user B is located, the first target data also includes a position identifier and a distance identifier, and updating the first video call screen specifically includes: updating the displayed first video call screen to the first target screen, and superimposing and displaying the first 3D model and the distance identifier of the target object at the first position of the first target screen, and superimposing and displaying the position identifier at the second position of the first target screen, the position identifier is used to indicate the display position of the target object in the first video call screen, and the distance identifier is used to indicate the distance between the target object and the second terminal.

[0068] Utilize SLAM technology to track the spatial position of the target object in real time. The first server continuously calculates the distance between the position of the second terminal's camera and the position of the target object, and displays it in real time through the distance marker; at the same time, a dynamic arrow will appear on the screen to guide the approximate position of the target object, that is, the display position of the target object in the first video call screen. In order to maintain the stable performance of the three-dimensional model in the video, the first server can also update the camera angle and position of the second terminal in real time, reposition the three-dimensional model, and ensure that its spatial position does not change. In this way, when the target object is not displayed within the camera range of the second terminal, the target object can still be quickly tracked through the position marker and distance marker.

[0069] The location marker includes at least one of the following: a static marker and a dynamic marker. Preferably, a dynamic marker is used for display, and using a dynamic marker, such as a flashing special effect, to indicate a target object can more effectively attract the user's attention and help them quickly locate the target object.

[0070] In some optional embodiments, the processing the first to-be-processed data to obtain first target data includes:

[0071] Acquire a reference image, wherein the reference image is an image of the target object in the first video call screen;

[0072] Generate a target image based on the reference image, wherein the target image is an image obtained after updating the target object from a first perspective to a second perspective, wherein the first perspective is a shooting perspective of the target object in the first video call screen;

[0073] Taking the reference image as a reference and the first viewing angle and the target image as constraints, optimizing the acquired initial neural radiation field model to obtain a target neural radiation field model;

[0074] Inputting the reference image into the target neural radiation field model for recognition to obtain texture information of the reference image;

[0075] Point cloud mapping is performed based on the texture information to obtain the first 3D model.

[0076] In this embodiment, first, the image corresponding to the target object in the first video call screen determined by user A based on the first terminal is used as a reference image, and a 2D diffusion model is used to generate a target image of a new perspective (ie, a second perspective) to enhance 3D perception.

[0077] Then, the obtained initial neural radiance field (NeRF) model is optimized, which can map 3D coordinates and viewing directions to colors and densities. Taking the reference image as the benchmark, the first view and the target image as constraints, the NeRF model is optimized by rendering a view that matches the reference image while ensuring consistency when observed from the second view. During the optimization process, the second view image generated by the diffusion model is introduced as prior information to assist NeRF in understanding possible 3D structures.

[0078] After obtaining the target neural radiation field model, the optimized target neural radiation field model is converted into a point cloud representation, which includes the position, color and probability of each point. The reference image is input into the target neural radiation field model for recognition, and the point cloud is texture mapped using the texture information in the reference image to increase detail and realism. The point cloud is further optimized to ensure that its visual effect at the new perspective is consistent with the diffusion prior. The point cloud is refined to add additional details so that the first 3D model obtained can show high-quality texture and geometric details at different perspectives.

[0079] Finally, the generated first 3D model is post-processed to remove noise and smooth it to improve the visual quality.

[0080] like Figure 4 As shown, the embodiment of the present application further provides a video interaction device, the video interaction device 400 includes:

[0081] A first receiving module 401 is configured to receive first data to be processed, wherein the first data to be processed includes a first video call screen and a target object specified by a user in the first video call screen, wherein the first video call screen is generated during a video call between a first terminal and a second terminal;

[0082] A first processing module 402 is configured to process the first data to be processed to obtain first target data, wherein the first target data includes a first target screen and a first 3D model of the target object, and the first target screen is a screen after the target object in the first video call screen is deleted;

[0083] The first sending module 403 is used to send the first target data to the first terminal and the second terminal, so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model.

[0084] Optionally, the video interaction device 400 further includes:

[0085] an updating module, configured to update the first 3D model in response to a first request received from the first terminal or the second terminal to obtain a second 3D model, wherein the first request is used to request an update of at least one of the following parameters of the first 3D model: a display position, a model posture, and a model size;

[0086] The second sending module is used to send the second 3D model to the first terminal and the second terminal, so that the first terminal and the second terminal update the displayed first 3D model.

[0087] Optionally, the video interaction device 400 further includes:

[0088] A second receiving module is used to receive second data to be processed, wherein the second data to be processed includes a second video call picture and a posture change amount of the camera of the second terminal, and the second video call picture is a video call picture generated according to a picture collected by the camera of the second terminal after the posture of the camera of the second terminal changes;

[0089] A second processing module is configured to process the second data to be processed and the first 3D model to obtain second target data, wherein the second target data includes a second target screen and a third 3D model of the target object, the second target screen is a screen after deleting the target object in the second video call screen, and the third 3D model is a 3D model obtained by updating the posture of the first 3D model based on the posture change amount;

[0090] The third sending module is used to send the second target data to the first terminal and the second terminal, so that the first terminal and the second terminal update the displayed video call screen according to the second target screen and the third 3D model.

[0091] Optionally, the first processing module 402 includes:

[0092] a deleting submodule, configured to delete the target object from the first video call screen according to the position information of the target object in the first video call screen, so as to obtain a first intermediate screen;

[0093] The repair submodule is used to perform image repair on the position where the target object is deleted in the first intermediate picture to obtain the first target picture.

[0094] Optionally, the first processing module 402 includes:

[0095] A first acquisition submodule, configured to acquire a reference image, wherein the reference image is an image of the target object in the first video call screen;

[0096] an updating submodule, configured to generate a target image based on the reference image, wherein the target image is an image obtained after updating the target object from a first perspective to a second perspective, wherein the first perspective is a shooting perspective of the target object in the first video call screen;

[0097] A second acquisition submodule is used to optimize the acquired initial neural radiation field model based on the reference image and the first viewing angle and the target image as constraints to obtain a target neural radiation field model;

[0098] A recognition submodule, used for inputting the reference image into the target neural radiation field model for recognition to obtain texture information of the reference image;

[0099] A mapping submodule is used to perform point cloud mapping based on the texture information to obtain the first 3D model.

[0100] Optionally, the first target data further includes a location identifier and a distance identifier, and updating the first video call screen specifically includes:

[0101] The displayed first video call screen is updated to the first target screen, and the first 3D model of the target object and the distance identifier are superimposed and displayed at the first position of the first target screen, and the position identifier is superimposed and displayed at the second position of the first target screen, wherein the position identifier is used to indicate the display position of the target object in the first video call screen, and the distance identifier is used to indicate the distance between the target object and the second terminal.

[0102] It should be noted that the video interaction device 400 provided in the embodiment of the present application can achieve the following Figure 1 The entire technical process of the video interaction method shown in the embodiment achieves the same technical effect and will not be described again here to avoid repetition.

[0103] The embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program is executed by the processor to implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0104] For details, see Figure 5 As shown, an embodiment of the present application further provides an electronic device, including a bus 501, a transceiver 502, an antenna 503, a bus interface 504, a processor 505 and a memory 506.

[0105] In this embodiment, the electronic device further includes: a computer program stored in the memory 506 and executable on the processor 505. When the computer program is executed by the processor 505, the following can be achieved: Figure 1 The various processes of the video interaction method shown in the embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0106] exist Figure 5In the embodiment, the bus architecture (represented by bus 501) is shown, and bus 501 may include any number of interconnected buses and bridges, and bus 501 links various circuits including one or more processors represented by processor 505 and memory represented by memory 506. Bus 501 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 504 provides an interface between bus 501 and transceiver 502. Transceiver 502 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 505 is transmitted on a wireless medium via antenna 503, and further, antenna 503 also receives data and transmits the data to processor 505.

[0107] The processor 505 is responsible for managing the bus 501 and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 506 can be used to store data used by the processor 505 when performing operations.

[0108] Optionally, the processor 505 may be a CPU, an ASIC, an FPGA or a CPLD.

[0109] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned video interaction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, ROM, RAM, a magnetic disk or an optical disk.

[0110] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0111] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0113] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video interaction method, characterized in that: The method comprises: Receiving first data to be processed, wherein the first data to be processed includes a first video call screen and a target object specified by a user in the first video call screen, and the first video call screen is generated during a video call between a first terminal and a second terminal; Processing the first data to be processed to obtain first target data, wherein the first target data includes a first target screen and a first 3D model of the target object, and the first target screen is a screen after deleting the target object in the first video call screen; The first target data is sent to the first terminal and the second terminal, so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model.

2. The method according to claim 1, characterized in that After sending the first target data to the first terminal and the second terminal so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model, the method further includes: In case of receiving a first request from the first terminal or the second terminal, in response to the first request, updating the first 3D model to obtain a second 3D model, wherein the first request is used to request updating at least one of the following parameters of the first 3D model: display position, model posture, and model size; The second 3D model is sent to the first terminal and the second terminal, so that the first terminal and the second terminal update the displayed first 3D model.

3. The method according to claim 1, characterized in that In a case where the first video call screen is a video call screen generated according to a screen captured by a camera of the second terminal during a video call between the first terminal and the second terminal, after sending the first target data to the first terminal and the second terminal so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model, the method further includes: receiving second data to be processed, wherein the second data to be processed includes a second video call screen and a posture change amount of a camera of the second terminal, and the second video call screen is a video call screen generated according to a screen captured by the camera of the second terminal after the posture of the camera of the second terminal changes; Processing the second data to be processed and the first 3D model to obtain second target data, wherein the second target data includes a second target screen and a third 3D model of the target object, the second target screen is a screen after deleting the target object in the second video call screen, and the third 3D model is a 3D model obtained by updating the posture of the first 3D model based on the posture change amount; The second target data is sent to the first terminal and the second terminal, so that the first terminal and the second terminal update the displayed video call screen according to the second target screen and the third 3D model.

4. The method according to claim 1, characterized in that The step of processing the first data to be processed to obtain first target data includes: Deleting the target object from the first video call screen according to the position information of the target object in the first video call screen to obtain a first intermediate screen; Image restoration is performed on the position in the first intermediate picture where the target object is deleted to obtain the first target picture.

5. The method according to claim 1, characterized in that The step of processing the first data to be processed to obtain first target data includes: Acquire a reference image, wherein the reference image is an image of the target object in the first video call screen; Generate a target image based on the reference image, wherein the target image is an image obtained after updating the target object from a first perspective to a second perspective, wherein the first perspective is a shooting perspective of the target object in the first video call screen; Taking the reference image as a reference and the first viewing angle and the target image as constraints, optimizing the acquired initial neural radiation field model to obtain a target neural radiation field model; Inputting the reference image into the target neural radiation field model for recognition to obtain texture information of the reference image; Point cloud mapping is performed based on the texture information to obtain the first 3D model.

6. The method according to claim 1, characterized in that The first target data also includes a location identifier and a distance identifier, and updating the first video call screen specifically includes: The displayed first video call screen is updated to the first target screen, and the first 3D model of the target object and the distance identifier are superimposed and displayed at the first position of the first target screen, and the position identifier is superimposed and displayed at the second position of the first target screen, wherein the position identifier is used to indicate the display position of the target object in the first video call screen, and the distance identifier is used to indicate the distance between the target object and the second terminal.

7. A video interaction device, characterized in that: include: A first receiving module, configured to receive first data to be processed, wherein the first data to be processed includes a first video call screen and a target object specified by a user in the first video call screen, wherein the first video call screen is generated during a video call between a first terminal and a second terminal; A first processing module is configured to process the first data to be processed to obtain first target data, wherein the first target data includes a first target screen and a first 3D model of the target object, and the first target screen is a screen after deleting the target object in the first video call screen; The first sending module is used to send the first target data to the first terminal and the second terminal, so that the first terminal and the second terminal update the first video call screen according to the first target screen and the first 3D model.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the video interaction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video interaction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the video interaction method as claimed in any one of claims 1 to 6.

Images