Remote cooperative interaction method and device and electronic equipment

Abstract

The invention discloses a remote cooperative interaction method and device and electronic equipment. Relates to the communication field, and the method comprises the following steps: receiving a cooperation establishment request sent by a field end, and obtaining an operation environment image sent by the field end, the cooperation establishment request being used for requesting to establish a remote cooperation task between the field end and a remote end; information to be coordinated sent by the field end is received, a remote guidance image is generated according to the information to be coordinated and the operation environment image, and the remote guidance image comprises image data of the operation action and image data of the operation object; the remote guidance image is sent to a field end, and the field end processes the operation object according to the image data of the operation action. According to the method and the device, the problem of relatively low accuracy and efficiency of remote guidance through a telephone or a video in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular, to a remote collaborative interaction method, device, and electronic device. Background Art

[0002] With the rapid development of smart manufacturing and digital society, remote collaboration has become increasingly important in multiple business scenarios, such as power equipment inspection and maintenance, emergency maintenance and training, etc. On-site personnel often lack the experience to accurately locate problems or solve them efficiently and safely, so remote supervision and guidance from experts are needed.

[0003] However, existing remote guidance usually uses telephone or video. When the guidance content is relatively complex, the use of remote guidance by telephone or video is likely to cause incomplete understanding or misoperation by on-site personnel, thereby affecting the normal handling of on-site business.

[0004] Currently, no effective solution has been proposed to address the low accuracy and efficiency of remote guidance via telephone or video in related technologies. Summary of the invention

[0005] The present application provides a remote collaborative interaction method, device and electronic device to solve the problem of low accuracy and efficiency of remote guidance via telephone or video in the related art.

[0006] According to one aspect of the present application, a remote collaborative interaction method is provided. The method includes: receiving a collaborative establishment request sent by a field end, and acquiring an operating environment image sent by the field end, wherein the collaborative establishment request is used to request the establishment of a remote collaborative task between the field end and the remote end; receiving information to be collaborated with sent by the field end, and generating a remote guidance image based on the information to be collaborated with and the operating environment image, wherein the remote guidance image includes image data of the operation action and image data of the operation object; sending the remote guidance image to the field end, wherein the field end processes the operation object based on the image data of the operation action.

[0007] Optionally, obtaining the working environment image sent by the on-site end includes: responding to a request to establish collaboration and establishing a data interaction link between the on-site end and the remote end; receiving a video stream sent by the on-site end according to the data interaction link, and reconstructing the working environment image of the on-site end according to the video stream, wherein the video stream is video information obtained by the on-site end by scanning the working environment.

[0008] Optionally, generating a remote guidance image based on the information to be collaborated and the working environment image includes: determining the collaboration content based on the information to be collaborated, and judging whether there is information about newly added operation objects in the collaboration content, wherein the newly added operation objects are objects that do not appear in the working environment image; if there is information about the newly added operation objects in the collaboration content, obtaining M initial two-dimensional images of the newly added operation objects based on the information about the newly added operation objects, generating a three-dimensional image of the newly added operation objects based on the M initial two-dimensional images, and generating a remote guidance image based on the three-dimensional image and the working environment image, wherein the viewing angle information of each initial two-dimensional image is different, and M is a positive integer; if there is no information about the newly added operation objects in the collaboration content, generating image data of operation information and operation actions in the working environment image based on the collaboration content, and determining the working environment image to which the image data of operation information and operation actions have been added as the remote guidance image.

[0009] Optionally, generating a three-dimensional image of a newly added operation object based on M initial two-dimensional images includes: stretching each initial two-dimensional image separately to obtain a three-dimensional image of each initial two-dimensional image; acquiring feature points in each three-dimensional image to obtain M initial feature point sets, and filtering the feature points in each initial feature point set to obtain M target feature point sets; projecting the feature points in each target feature point set onto a two-dimensional plane to obtain a target two-dimensional image corresponding to each initial two-dimensional image; inputting each target two-dimensional image into a three-dimensional image generation network to obtain a three-dimensional image corresponding to each target two-dimensional image; and synthesizing the M three-dimensional images according to the viewing angle information to obtain a three-dimensional image of the newly added operation object.

[0010] Optionally, generating a remote guidance image based on a three-dimensional image and a working environment image includes: generating image data of operating actions in the working environment image based on the collaborative content and the three-dimensional image, and generating operation information corresponding to the operating actions; and determining the working environment image to which the operation information and image data of the operating actions have been added as the remote guidance image.

[0011] Optionally, the working environment image includes a person image. After acquiring the working environment image sent by the on-site end, the method further includes: identifying the working area in the working environment image; determining whether the person image is located in the working area; if the person image is located in the working area, determining that there is no abnormality in the working environment image; if the person image is not located in the working area, determining that there is an abnormality in the working environment image, and generating prompt information based on the position information of the person image in the working environment image, and sending the prompt information to the on-site end.

[0012] Optionally, sending the remote guidance image to the on-site end includes: sending the remote guidance image to a cloud server, storing the remote guidance image in a blockchain to which the remote end belongs through the cloud server, and sending the remote guidance image to the on-site end through the cloud server, wherein the cloud server is also used to store the collaboration establishment request sent by the on-site end.

[0013] According to another aspect of the present application, a remote collaborative interaction device is provided. The device includes: a receiving unit, used to receive a collaborative establishment request sent by a field end, and obtain an operating environment image sent by the field end, wherein the collaborative establishment request is used to request the establishment of a remote collaborative task between the field end and the remote end; a generating unit, used to receive information to be collaborated with sent by the field end, and generate a remote guidance image based on the information to be collaborated with and the operating environment image, wherein the remote guidance image includes image data of the operation action and image data of the operation object; a sending unit, used to send the remote guidance image to the field end, wherein the field end processes the operation object based on the image data of the operation action.

[0014] According to another aspect of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, a remote collaborative interaction method provided in the aforementioned embodiment of the present application is implemented.

[0015] According to another aspect of the present invention, an electronic device is also provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein when the computer-readable instructions are executed, a remote collaborative interaction method provided in the aforementioned embodiment is executed.

[0016] Through this application, the following steps are adopted: receiving a collaboration establishment request sent by the on-site end, and obtaining the working environment image sent by the on-site end, wherein the collaboration establishment request is used to request the establishment of a remote collaboration task between the on-site end and the remote end; receiving the information to be collaborated sent by the on-site end, and generating a remote guidance image according to the information to be collaborated and the working environment image, wherein the remote guidance image contains the image data of the operation action and the image data of the operation object; sending the remote guidance image to the on-site end, wherein the on-site end processes the operation object according to the image data of the operation action. The problem of low accuracy and efficiency of remote guidance through telephone or video in the related technology is solved. By establishing a remote collaboration task between the on-site end and the remote end according to the working environment image sent by the on-site end, and generating a guidance image according to the information to be collaborated, and then sending the guidance image to the on-site end, the on-site end and the remote end can perform remote guidance operations through the remote guidance image by generating the remote guidance image based on the synchronized working environment image, thereby achieving the effect of improving the accuracy and efficiency of the remote guidance operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a flowchart of a remote collaborative interaction method provided according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a three-dimensional image generation process provided according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a remote collaborative interaction system provided according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a remote collaborative interaction device provided according to an embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the remote collaborative interaction method, device and electronic device determined in the present disclosure can be used in the communication field, and can also be used in any field other than the communication field. The application field of the remote collaborative interaction method, device and electronic device determined in the present disclosure is not limited.

[0027] It should be noted that the collected information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) used in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated. Corresponding operation portals are provided for users to choose to authorize or refuse to use. For example, an interface is set up between this system and relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0028] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0029] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0030] AR device: (Augmented Reality Device), through AR technology, virtual information can be superimposed on the real world, allowing users to see virtual objects, text or images and interact with the real environment.

[0031] VR device: (Virtual Reality Device) is a simulation scene simulated by computer technology, which allows users to immerse themselves in and interact with the virtual world. Virtual reality devices are hardware devices used to experience virtual reality technology, including head-mounted displays, handles, sensors, etc. Through these devices, users can experience an immersive virtual world experience.

[0032] RGB frame: "Red Green Blue Frame". RGB is a color mode that uses a combination of red, green and blue to represent various colors. In video and image processing, an RGB frame is an image frame composed of three color channels: red, green and blue.

[0033] According to an embodiment of the present application, a remote collaborative interaction method is provided.

[0034] Figure 1 1 is a flow chart of a remote collaborative interaction method provided according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0035] Step S101 , receiving a collaboration establishment request sent by a field end, and acquiring a working environment image sent by the field end, wherein the collaboration establishment request is used to request to establish a remote collaboration task between the field end and the remote end.

[0036] It should be noted that the remote end can be a VR device worn by a remote expert who guides the local workers, such as a VR headset, and the on-site end can be an AR device worn by the local workers, such as an AR headset. The on-site end device can synchronize the working environment image with the remote end by scanning the working environment, so that the remote end can simulate the on-site working environment through the device.

[0037] Specifically, when the on-site end needs remote assistance from the remote end, a collaboration establishment request can be sent to the remote end. After receiving the collaboration establishment request, the remote end can establish a data synchronization channel between the on-site end and the remote end according to the collaboration establishment request, so that the remote end can receive the working environment image sent by the on-site end, and then can synchronously obtain the working environment image of the on-site end in the remote end device.

[0038] Furthermore, after the data synchronization channel is established, data interaction can be carried out between the remote end and the on-site end, thereby performing interactive operations of guidance information based on the working environment image.

[0039] For example, a remote expert can use a VR device on the remote end to observe a shared 3D local scene and create instructions in the VR space. A computer device can be used to run a VR client and a VR headset with a hand tracker fixed in front. The remote user can freely navigate and observe the 3D scene shared from the local end by wearing the VR headset. To support natural gesture interaction, the client uses a hand tracker to track the hand in real time, allowing the remote expert to create instructions using gestures, which will then be transmitted to the local end to guide the local field operator.

[0040] Further, the local operator receives and follows the instructions to perform the task. A computer device can be used to run an AR client, an AR headset, a hand tracker fixed in front, and a depth camera fixed above the workspace. The local operator can see the virtual instructions superimposed on the real scene from the remote end by wearing the AR headset, instructing the local operator how to perform physical tasks. The client also uses the hand tracker to track the hands in real time to support real-time visual notification of the interaction between the local user's body (i.e., hands and arms) and the annotated three-dimensional space area. The depth camera supports real-time high-resolution capture of the entire local workspace, and then the three-dimensional video composed of depth and RGB frames is shared to the remote expert end to reconstruct the three-dimensional stereo scene. In order to unify the coordinate system of the virtual space and the real world space, an artificial marker can be set in the local workspace as a unified benchmark to ensure the accuracy of the image.

[0041] Step S102, receiving the information to be coordinated sent by the on-site end, and generating a remote guidance image according to the information to be coordinated and the working environment image, wherein the remote guidance image includes the image data of the operation action and the image data of the operation object.

[0042] Specifically, after establishing communication between the remote end and the on-site end, the on-site end can send information to be collaborated with to the remote end, thereby sending issues requiring collaboration to the remote end through the information to be collaborated with. After receiving the information to be collaborated with, the remote end can generate corresponding collaboration information based on the information to be collaborated with, wherein the collaboration information is used to guide the on-site personnel to perform business processing, and further generate corresponding guidance images in the environmental operation images based on the collaboration information, so that the on-site operating personnel can process the business based on the remote guidance images.

[0043] It should be noted that the remote guidance image contains image data of the operating action and image data of the operating object. That is, it is necessary to generate image data of the operating object to be operated in the environmental work image, and generate corresponding image data of the operating action, so that the operating personnel can process the on-site business according to the image data of the operating action and the image data of the operating object.

[0044] For example, the local operator shares and transmits the 3D video stream to the remote VR terminal. On the remote VR terminal, the remote VR terminal receives and reconstructs the real-time 3D scene of the local workspace through the 3D video stream. Then, the remote expert obtains the real-time situation of the local workspace through this 3D reconstruction technology to improve situational awareness. Then, the remote expert can create guidance information through simple and natural interactions using gestures to create 3D sketches and 3D spatial area prompts. These non-verbal communication prompts will be shared with the local AR terminal in real time and superimposed on the real local workspace, thereby obtaining remote guidance images, and synchronizing the remote guidance images to the on-site terminal.

[0045] Step S103, sending the remote guidance image to the on-site end, wherein the on-site end processes the operation object according to the image data of the operation action.

[0046] It should be noted that after the remote guidance image is generated, the remote guidance image can be synchronized to the on-site end, so that the on-site end can process the real operation object on site according to the image data of the operation action in the remote guidance image and the image data of the operation object, thereby completing the remote guidance operation process and improving the accuracy and efficiency of remote guidance.

[0047] For example, the visual cues received by the AR end are superimposed on the real working environment of the local field operator. The local field operator sees the remote guidance image through the AR headset and performs corresponding operations locally based on the remote guidance image, language and other information.

[0048] The remote collaborative interaction method provided by the embodiment of the present application receives a collaborative establishment request sent by the on-site end, and obtains the working environment image sent by the on-site end, wherein the collaborative establishment request is used to request the establishment of a remote collaborative task between the on-site end and the remote end; receives the information to be collaborated sent by the on-site end, and generates a remote guidance image according to the information to be collaborated and the working environment image, wherein the remote guidance image contains the image data of the operation action and the image data of the operation object; sends the remote guidance image to the on-site end, wherein the on-site end processes the operation object according to the image data of the operation action. The problem of low accuracy and efficiency of remote guidance through telephone or video in the related technology is solved. By establishing a remote collaborative task between the on-site end and the remote end according to the working environment image sent by the on-site end, and generating a guidance image according to the information to be collaborated, and then sending the guidance image to the on-site end, the on-site end and the remote end can perform remote guidance operations through the remote guidance image by generating the remote guidance image based on the synchronized working environment image, thereby achieving the effect of improving the accuracy and efficiency of the remote guidance operation.

[0049] Optionally, in the remote collaborative interaction method provided in an embodiment of the present application, obtaining the working environment image sent by the on-site end includes: responding to a collaboration establishment request and establishing a data interaction link between the on-site end and the remote end; receiving a video stream sent by the on-site end according to the data interaction link, and reconstructing the working environment image of the on-site end based on the video stream, wherein the video stream is video information obtained by the on-site end by scanning the working environment.

[0050] Specifically, when acquiring the working environment image sent by the on-site end, it is first necessary to respond to the collaboration establishment request, so as to establish a data interaction link between the on-site end and the remote end, and then complete the data interaction operation according to the data interaction link. Furthermore, after obtaining the data interaction link, the on-site end can use the data interaction link to transmit the on-site image, that is, send the video stream of the on-site image. After receiving the video stream, the remote end can reconstruct the working environment image of the on-site end according to the video stream, so that the remote end and the on-site end can share the working environment image.

[0051] For example, operators on-site can add or edit tasks, define in detail information such as task objectives, required resources, and participants, and initiate collaboration requests. Through the space sharing function, remote users can share work environment images with on-site users and conduct real-time audio and video communications, thereby completing the transmission of work environment images.

[0052] This embodiment ensures normal transmission of the working environment image by establishing a data interaction link between the on-site end and the remote end.

[0053] Optionally, in the remote collaborative interaction method provided in the embodiment of the present application, generating a remote guidance image based on the information to be collaborated and the working environment image includes: determining the collaborative content based on the information to be collaborated, and judging whether there is information about a newly added operation object in the collaborative content, wherein the newly added operation object is an object that does not appear in the working environment image; if there is information about the newly added operation object in the collaborative content, obtaining M initial two-dimensional images of the newly added operation object based on the information about the newly added operation object, generating a three-dimensional image of the newly added operation object based on the M initial two-dimensional images, and generating a remote guidance image based on the three-dimensional image and the working environment image, wherein the viewing angle information of each initial two-dimensional image is different, and M is a positive integer; if there is no information about the newly added operation object in the collaborative content, generating image data of the operation information and the operation action in the working environment image based on the collaborative content, and determining the working environment image to which the operation information and the image data of the operation action have been added as the remote guidance image.

[0054] Specifically, when generating a remote guidance image, it is first necessary to obtain the collaboration content in the information to be collaborated, so as to determine whether there is relevant information of a new operation object, where the new operation object is whether a new object needs to be added under the premise of the current working environment. For example, the current working environment is to assemble a certain device. If a screw needs to be added to the device, the screw is also a new operation object.

[0055] Further, in the case where it is determined that there is a newly added operation object, since there is no image of the newly added operation object in the working environment image, the remote end needs to generate a three-dimensional image of the newly added operation object based on the information of the newly added operation object, and then remote guidance information can be generated based on the three-dimensional image. Wherein, when generating the remote guidance information, it is first necessary to obtain M initial two-dimensional images of the newly added operation object, and then generate a three-dimensional image of the newly added operation object based on the M initial two-dimensional images, wherein each initial two-dimensional image is a two-dimensional image of the newly added operation object at a different viewing angle, such as a front view, a top view, a side view, etc., and then the three-dimensional image of the newly added operation object is obtained by combining the M initial two-dimensional images.

[0056] After obtaining the three-dimensional image of the newly added operation object, an operation image for processing the newly added operation object can be generated, and the operation process of the operation image and the three-dimensional image of the newly added operation object in the working environment image is recorded to obtain a remote guidance image.

[0057] It should be noted that after generating a three-dimensional image of the newly added operation object, the operation image after the three-dimensional image of the newly added operation object in the working environment image can be synchronized to the on-site end in real time, so that the on-site end can obtain the remote guidance image in time and perform corresponding operations according to the remote guidance image.

[0058] It should be noted that, when there is no information about the newly added operation object in the collaborative content, it is not necessary to generate a three-dimensional image of the newly added operation object. At this time, the operation image can be directly generated in the working environment image. At this time, the operation object can be a device object existing in the working environment image. At the same time, operation information, such as prompt information, etc., can also be generated, and the working environment image to which the operation information and image data of the operation action have been added is determined as a remote guidance image, and synchronized to the on-site end to complete the guidance process of the corresponding operation.

[0059] For example, there is a device in the working environment image, the operation object can be the device, and the operation action can be the action performed on the device, such as clicking a button. At this time, the operation action can be added to the working environment image, that is, the operation action is added to the working environment image, and then a remote guidance image is obtained.

[0060] This embodiment achieves the technical effect of accurately generating remote guidance images by combining the operation action, the image of the operation object and the image of the working environment.

[0061] Optionally, in the remote collaborative interaction method provided in the embodiment of the present application, generating a three-dimensional image of a newly added operation object based on M initial two-dimensional images includes: stretching each initial two-dimensional image separately to obtain a three-dimensional image of each initial two-dimensional image; acquiring feature points in each three-dimensional image to obtain M initial feature point sets, and filtering the feature points in each initial feature point set to obtain M target feature point sets; projecting the feature points in each target feature point set onto a two-dimensional plane to obtain a target two-dimensional image corresponding to each initial two-dimensional image; inputting each target two-dimensional image into a three-dimensional image generation network to obtain a three-dimensional image corresponding to each target two-dimensional image; and synthesizing the M three-dimensional images according to the viewing angle information to obtain a three-dimensional image of the newly added operation object.

[0062] Specifically, when generating a three-dimensional image of a newly added operation object, it is first necessary to obtain M initial two-dimensional images of the newly added operation object, and perform stretching operations on each of the two-dimensional images to obtain a three-dimensional image corresponding to each of the two-dimensional images. For example, an initial two-dimensional sketch of the newly added operation object at various angles is drawn in a drawing software, and then a preliminary three-dimensional image at various angles is shaped through other geometric operations such as "stretching, rotating or sweeping".

[0063] For each three-dimensional image, feature points of the three-dimensional image can be obtained, filtered, and projected onto a two-dimensional plane through the filtered feature points, thereby obtaining a target two-dimensional image that is more accurate than the initial two-dimensional image.

[0064] For example, in the 3D modeling process, the initial 3D point set of each 3D image is obtained, and the 3D points contain feature points. The feature points in the point set may contain outliers due to various reasons (for example, noise or measurement errors in the data capture process). At this time, a random sampling consensus algorithm can be applied to identify and remove these outliers (i.e., discrete points that deviate from the expected plane). This algorithm finds the best fitting geometric structure (in this case, a two-dimensional plane) by repeatedly sampling several points in the data set. After removing the outliers, the least squares algorithm can be used to fit the remaining internal points to an optimal two-dimensional plane, thereby obtaining an optimal plane representing the point set, so as to facilitate the continued transformation and application of the plane. Finally, each 3D sketch point (i.e., point set) is projected onto the fitted two-dimensional plane to form a point set of a pure two-dimensional shape, thereby obtaining an accurate target two-dimensional image.

[0065] Through the above steps, the following technical effects can be achieved:

[0066] Improve data quality: When drawing 2D sketches, especially when acquiring data from hand sketches or noisy data, outliers or abnormal points may be introduced. These points may not conform to the expected model or shape and may cause inaccurate 3D reconstruction. We can focus on identifying and ignoring these abnormal points by applying a random sampling consensus algorithm.

[0067] Enhance model robustness: Applying the random sampling consensus algorithm is an iterative algorithm that works by finding the best fitting model for the data while considering the minimum number of data points. It can effectively handle data with a large number of outliers, making the resulting model more reliable.

[0068] Improve fitting accuracy: After outliers are removed, the least squares algorithm can perform the best plane fit on the remaining inliers. This combination can fit the 2D plane more accurately, providing a more accurate basis for subsequent 3D operations.

[0069] Optimizing computational efficiency: After removing outlier data points, the data set becomes smaller and more precise, which can reduce the computational burden and improve the efficiency and accuracy of subsequent processing steps.

[0070] Furthermore, after obtaining M target two-dimensional images, the M target two-dimensional images can be input into a three-dimensional image generation network to obtain three-dimensional images corresponding to each target two-dimensional image, wherein the three-dimensional image generation network can be a neural network model obtained by training an initial neural network model using multiple two-dimensional images and each two-dimensional image corresponding to a three-dimensional image as samples, and the three-dimensional image generation network can generate corresponding three-dimensional images according to each feature point in the two-dimensional image.

[0071] Finally, after obtaining the 3D images corresponding to the 2D images, since different 2D images have different viewing angles, the M 3D images can be synthesized according to the viewing angle information to obtain the 3D image of the newly added operation object.

[0072] Figure 2 is a flow chart of a three-dimensional image generation process provided in an embodiment of the present application, such as Figure 2As shown in the figure, first draw a sketch in three-dimensional space, and the sketch creation process will be synchronized to the local workspace in real time, so that local staff can see the operation of the remote expert and the generation of the sketch. The drawn sketch is further processed by the random sampling consistency algorithm to remove abnormal points (such as discrete points that deviate from the sketch plane), and then the least squares algorithm is used to fit the internal points to the two-dimensional plane to obtain a two-dimensional contour, that is, a two-dimensional image. After that, the two-dimensional image is used as input and combined with the known viewpoint information through the three-dimensional generation network to generate a three-dimensional model of each two-dimensional image, and the three-dimensional image is synthesized through the three-dimensional image synthesis model to obtain the three-dimensional image of the newly added operation object.

[0073] This embodiment achieves the technical effect of accurately generating a three-dimensional image of the newly added operation object by processing the two-dimensional image of the newly added operation object.

[0074] Optionally, in the remote collaborative interaction method provided in an embodiment of the present application, generating a remote guidance image based on a three-dimensional image and a working environment image includes: generating image data of operating actions in the working environment image based on the collaborative content and the three-dimensional image, and generating operation information corresponding to the operating actions; and determining the working environment image to which the operation information and image data of the operating actions have been added as the remote guidance image.

[0075] Specifically, after obtaining the three-dimensional image, image data of the operation action can be generated based on the three-dimensional image, and the image data of the operation action is the operation of processing the three-dimensional image of the newly added operation object. After obtaining the image data of the operation action, operation information corresponding to the operation action can be generated, so as to further explain the operation action according to the operation information.

[0076] Furthermore, after obtaining the image data and operation information of the operation action, the image data and operation information of the operation action can be fused with the working environment image, and then the working environment image to which the operation information and image data of the operation action have been added can be determined as the remote guidance image.

[0077] Optionally, in the remote collaborative interaction method provided in the embodiment of the present application, the working environment image includes a person image. After obtaining the working environment image sent by the on-site end, the method also includes: identifying the working area in the working environment image; determining whether the person image is located in the working area; if the person image is located in the working area, determining that there is no abnormality in the working environment image; if the person image is not located in the working area, determining that there is an abnormality in the working environment image, and generating prompt information based on the position information of the person image in the working environment image, and sending the prompt information to the on-site end.

[0078] Specifically, there may be a work area in the work environment image. The remote end not only needs to guide the work process, but also needs to monitor the work position of the operator. First, it is necessary to identify the work area in the work environment image and determine whether the person image on the site is located in the work area. If the person image on the site is not located in the work area, it indicates that the operator on the site may be in a dangerous position or a prohibited position. At this time, prompt information can be generated based on the position information of the person image in the work environment image, so as to prompt the operator on the site to adjust the position in time to avoid work risks.

[0079] For example, the hand and body position of the local field operator can be continuously monitored. When the local field operator's body (i.e., hand or arm) enters or is about to enter the spatial area annotated by the three-dimensional spatial area prompt (i.e., risk area), the three-dimensional spatial area prompt will visually remind the remote expert and the local field operator by changing color and transparency. The local field operator can spontaneously adjust the operation based on the visual notification to avoid risky operations. At the same time, the remote expert can also remind the local field operator through verbal communication based on the visual notification. In this way, the local field operator can complete the physical task step by step safely and efficiently under the guidance of the remote expert.

[0080] This embodiment ensures the normal progress of on-site operations by identifying the positional relationship between the image of the operator and the operation area.

[0081] Optionally, in the remote collaborative interaction method provided in an embodiment of the present application, sending the remote guidance image to the on-site end includes: sending the remote guidance image to a cloud server, and storing the remote guidance image in a blockchain to which the remote end belongs through the cloud server, and sending the remote guidance image to the on-site end through the cloud server, wherein the cloud server is also used to store the collaboration establishment request sent by the on-site end.

[0082] Specifically, the field device and the remote device can be connected through the cloud server, that is, the field end and the remote end are connected to the cloud server respectively, and the data synchronization and data transmission between the field device and the remote device are completed through the cloud server.

[0083] It should be noted that the transmitted data can also be stored in the blockchain associated with the cloud server. By storing the data on the blockchain, the security and non-tamperability of the data can be ensured, which is convenient for future reference and auditing. By recording and analyzing the collaborative data, it can provide experience accumulation for subsequent tasks and improve the overall level of collaboration.

[0084] Figure 3 is a schematic diagram of a remote collaborative interaction system provided according to an embodiment of the present application, such as Figure 3As shown, the aforementioned remote collaborative interaction method is executed with an optional remote collaborative interaction system as the execution subject, and the remote collaborative interaction system at least includes: a cloud server 31, a remote terminal 32, a local terminal 33 (i.e., a field terminal) and a blockchain 34, wherein:

[0085] The remote end 32 can support remote experts to observe the shared 3D stereo local scene and create instructions in the immersive VR space. The remote end 32 consists of a computer device running a VR client and a VR headset. A hand tracker is fixed in front of the VR headset. The remote user can freely navigate and observe the three-dimensional stereo scene shared from the local end 33 by wearing the VR headset. To support natural gesture interaction, the client uses a hand tracker to track the hand in real time, allowing the remote expert to create instructions (i.e., three-dimensional sketches and three-dimensional space area prompts) using gestures. The instructions will then be transmitted to the local end 33 to guide the local field operator.

[0086] The local end 33 can support the local operator to receive and follow AR instructions to perform tasks. Specifically, the local end 33 can be composed of a computer device running an AR client, an AR headset, a hand tracker fixed in front of the AR headset, and a depth camera fixed above the workspace. The local user can see the virtual instructions superimposed on the real scene from the remote end 32 by wearing the AR headset, guiding them how to perform physical tasks. The client also uses the hand tracker to track the hands in real time to support real-time visual notification of the interaction between the local user's body (i.e., hands and arms) and the annotated three-dimensional space area. The depth camera supports real-time high-resolution capture of the entire local workspace, and then the three-dimensional video composed of RGB frames and depth frames is shared to the remote expert end to reconstruct the three-dimensional stereo scene. In order to unify the coordinate system of the virtual space and the real world space, an artificial marker can be set in the local workspace as a unified reference.

[0087] As the core hub of the entire platform, the cloud server 31 connects the remote VR end and the local AR end, and is responsible for managing and synchronizing various resources during the task to ensure the real-time and stability of collaboration. Specifically, the cloud server 31 is connected to the remote end 32 and the local AR end to ensure real-time synchronization of data at both ends. The cloud server 31 is responsible for managing and synchronizing three-dimensional video streams, as well as data channels (for transmitting gesture prompts, sketch data, model data, and visual notifications). The efficient transmission and processing of these data ensures seamless collaboration between remote users and local operators.

[0088] In addition, the remote collaborative interaction system can also integrate a variety of external systems through the cloud server 31, including a fund security inspection system, a work order system, a customer service system, a command and dispatch system, an asset management system, and a project management system. The cloud server 31 also has powerful spatial computing capabilities and can process and analyze large amounts of spatial data. This enables the platform to achieve complex spatial interactions and data visualization, providing users with a more intuitive and efficient collaborative experience. For example, the server can calculate changes in three-dimensional models in real time, support collaborative operations of multiple users in the same space, and provide accurate positioning and navigation functions. Through the above functions, the cloud server 31 effectively connects the remote VR end and the local AR end, integrates various external systems, provides powerful data processing and spatial computing capabilities, and builds a remote collaborative interaction system with complete functions and efficient collaboration.

[0089] Blockchain 34 is used to record the key steps of remote guidance and local operations in the form of photos or other data. Specifically, the recorded data of the collaboration process is stored on blockchain 34 to ensure that the data cannot be tampered with. Through blockchain 34 technology, the platform can provide a reliable traceability mechanism, allowing relevant parties to view and verify the entire collaboration process when necessary. The distributed characteristics of blockchain 34 are used to enhance the data security and reliability of the entire platform.

[0090] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0091] The embodiment of the present application also provides a remote collaborative interaction device. It should be noted that the remote collaborative interaction device of the embodiment of the present application can be used to execute the remote collaborative interaction method provided by the embodiment of the present application. The remote collaborative interaction device provided by the embodiment of the present application is introduced below.

[0092] Figure 4 Schematic diagram of a remote collaborative interaction device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0093] The receiving unit 41 is used to receive a collaboration establishment request sent by the on-site end and obtain the working environment image sent by the on-site end, wherein the collaboration establishment request is used to request to establish a remote collaboration task between the on-site end and the remote end.

[0094] The generating unit 42 is used to receive the information to be coordinated sent by the on-site end, and generate a remote guidance image according to the information to be coordinated and the working environment image, wherein the remote guidance image includes the image data of the operation action and the image data of the operation object.

[0095] The sending unit 43 is used to send the remote guidance image to the on-site end, wherein the on-site end processes the operation object according to the image data of the operation action.

[0096] The remote collaborative interaction device provided in the embodiment of the present application receives the collaborative establishment request sent by the on-site end through the receiving unit 41, and obtains the working environment image sent by the on-site end, wherein the collaborative establishment request is used to request the establishment of a remote collaborative task between the on-site end and the remote end; the generating unit 42 receives the information to be collaborated sent by the on-site end, and generates a remote guidance image according to the information to be collaborated and the working environment image, wherein the remote guidance image contains the image data of the operation action and the image data of the operation object; the sending unit 43 sends the remote guidance image to the on-site end, wherein the on-site end processes the operation object according to the image data of the operation action. The problem of low accuracy and efficiency of remote guidance through telephone or video in the related technology is solved. By establishing a remote collaborative task between the on-site end and the remote end according to the working environment image sent by the on-site end, generating a guidance image according to the information to be collaborated, and then sending the guidance image to the on-site end, the on-site end and the remote end can perform remote guidance operations through the remote guidance image by generating the remote guidance image based on the synchronized working environment image, thereby achieving the effect of improving the accuracy and efficiency of the remote guidance operation.

[0097] Optionally, in the remote collaborative interaction device provided in an embodiment of the present application, the receiving unit 41 includes: a response module, used to respond to a collaboration establishment request and establish a data interaction link between the on-site end and the remote end; a receiving module, used to receive a video stream sent by the on-site end according to the data interaction link, and reconstruct the working environment image of the on-site end based on the video stream, wherein the video stream is video information obtained by the on-site end by scanning the working environment.

[0098] Optionally, in the remote collaborative interaction device provided in the embodiment of the present application, the generation unit 42 includes: a first determination module, which is used to determine the collaborative content based on the information to be collaborated, and to determine whether there is information about a newly added operation object in the collaborative content, wherein the newly added operation object is an object that does not appear in the working environment image; a first acquisition module, which is used to obtain M initial two-dimensional images of the newly added operation object based on the information about the newly added operation object when there is information about the newly added operation object in the collaborative content, and to generate a three-dimensional image of the newly added operation object based on the M initial two-dimensional images, and to generate a remote guidance image based on the three-dimensional image and the working environment image, wherein the viewing angle information of each initial two-dimensional image is different, and M is a positive integer; a first generation module, which is used to generate image data of operation information and operation actions in the working environment image based on the collaborative content when there is no information about the newly added operation object in the collaborative content, and to determine the working environment image to which the image data of the operation information and operation actions has been added as the remote guidance image.

[0099] Optionally, in the remote collaborative interaction device provided in the embodiment of the present application, the first acquisition module includes: a processing module, which is used to stretch each initial two-dimensional image respectively to obtain a three-dimensional image of each initial two-dimensional image; a second acquisition module, which is used to acquire feature points in each three-dimensional image to obtain M initial feature point sets, and filter the feature points in each initial feature point set to obtain M target feature point sets; a projection module, which is used to project the feature points in each target feature point set onto a two-dimensional plane to obtain a target two-dimensional image corresponding to each initial two-dimensional image; an input module, which is used to input each target two-dimensional image into a three-dimensional image generation network to obtain a three-dimensional image corresponding to each target two-dimensional image; and a synthesis module, which is used to synthesize the M three-dimensional images according to the viewing angle information to obtain a three-dimensional image of the newly added operation object.

[0100] Optionally, in the remote collaborative interaction device provided in an embodiment of the present application, the first acquisition module includes: a second generation module, which is used to generate image data of operation actions in the working environment image based on the collaborative content and the three-dimensional image, and generate operation information corresponding to the operation actions; a second determination module, which is used to determine the working environment image to which the operation information and image data of the operation actions have been added as the remote guidance image.

[0101] Optionally, in the remote collaborative interaction device provided in the embodiment of the present application, the working environment image includes a person image. After obtaining the working environment image sent by the on-site end, the device also includes: an identification unit, used to identify the working area in the working environment image; a judgment unit, used to judge whether the person image is located in the working area; a first determination unit, used to determine that there is no abnormality in the working environment image when the person image is located in the working area; a second determination unit, used to determine that there is an abnormality in the working environment image when the person image is not located in the working area, and generate prompt information based on the position information of the person image in the working environment image, and send the prompt information to the on-site end.

[0102] Optionally, in the remote collaborative interaction device provided in an embodiment of the present application, the sending unit 43 includes: a sending module, used to send the remote guidance image to the cloud server, and store the remote guidance image in the blockchain to which the remote end belongs through the cloud server, and send the remote guidance image to the on-site end through the cloud server, wherein the cloud server is also used to store the collaboration establishment request sent by the on-site end.

[0103] The remote collaboration interaction device provided in the embodiment of the present application establishes a remote collaboration task between the on-site end and the remote end based on the working environment image sent by the on-site end, generates a guidance image based on the information to be collaborated, and then sends the guidance image to the on-site end, thereby generating a remote guidance image based on the synchronized working environment image, so that the on-site end and the remote end can perform remote guidance operations through the remote guidance image, thereby solving the problem of low accuracy and efficiency of remote guidance through telephone or video in related technologies, thereby achieving the effect of improving the accuracy and efficiency of remote guidance operations.

[0104] The above-mentioned remote collaborative interaction device includes a processor and a memory. The above-mentioned receiving unit 41, generating unit 42, sending unit 43, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0105] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of low accuracy and efficiency of remote guidance through telephone or video in related technologies can be solved by adjusting kernel parameters, thereby achieving the effect of improving the accuracy and efficiency of remote guidance operations.

[0106] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0107] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the remote collaborative interaction method is implemented.

[0108] An embodiment of the present invention provides a processor, which is used to run a program, wherein the remote collaborative interaction method is executed when the program is running.

[0109] Figure 5 is a schematic diagram of an electronic device provided according to an embodiment of the present application, such as Figure 5 As shown, an embodiment of the present invention provides an electronic device, the electronic device 50 includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: receiving a collaboration establishment request sent by the on-site end, and acquiring an operating environment image sent by the on-site end, wherein the collaboration establishment request is used to request the establishment of a remote collaboration task between the on-site end and the remote end; receiving information to be collaborated sent by the on-site end, and generating a remote guidance image based on the information to be collaborated and the operating environment image, wherein the remote guidance image includes image data of the operation action and image data of the operation object; sending the remote guidance image to the on-site end, wherein the on-site end processes the operation object based on the image data of the operation action. The device in this article can be a server, PC, PAD, mobile phone, etc.

[0110] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: receiving a collaboration establishment request sent by an on-site end, and obtaining an operating environment image sent by the on-site end, wherein the collaboration establishment request is used to request the establishment of a remote collaboration task between the on-site end and the remote end; receiving information to be collaborated with sent by the on-site end, and generating a remote guidance image based on the information to be collaborated with and the operating environment image, wherein the remote guidance image includes image data of the operation action and image data of the operation object; sending the remote guidance image to the on-site end, wherein the on-site end processes the operation object based on the image data of the operation action.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0112] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0117] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0118] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0119] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A remote collaborative interaction method, characterized in that: include: Receiving a collaboration establishment request sent by a field end, and acquiring a working environment image sent by the field end, wherein the collaboration establishment request is used to request to establish a remote collaboration task between the field end and a remote end; receiving the information to be coordinated sent by the on-site end, and generating a remote guidance image according to the information to be coordinated and the working environment image, wherein the remote guidance image includes image data of the operation action and image data of the operation object; The remote guidance image is sent to the on-site end, wherein the on-site end processes the operation object according to the image data of the operation action.

2. The method according to claim 1, characterized in that Acquiring the working environment image sent by the field end includes: Responding to the collaboration establishment request, and establishing a data interaction link between the on-site end and the remote end; The video stream sent by the field end is received according to the data interaction link, and the working environment image of the field end is reconstructed according to the video stream, wherein the video stream is the video information obtained by the field end by scanning the working environment.

3. The method according to claim 1, characterized in that Generating a remote guidance image according to the information to be collaborated and the working environment image includes: Determine the content of the collaboration according to the information to be collaborated, and judge whether there is information of a newly added operation object in the collaboration content, wherein the newly added operation object is an object that does not appear in the working environment image; In the case where the information of the newly added operation object exists in the collaboration content, M initial two-dimensional images of the newly added operation object are obtained according to the information of the newly added operation object, and a three-dimensional image of the newly added operation object is generated according to the M initial two-dimensional images, and the remote guidance image is generated according to the three-dimensional image and the working environment image, wherein the viewing angle information of each initial two-dimensional image is different, and M is a positive integer; When the information of the newly added operation object does not exist in the collaborative content, the operation information and the image data of the operation action are generated in the working environment image according to the collaborative content, and the working environment image to which the operation information and the image data of the operation action have been added is determined as the remote guidance image.

4. The method according to claim 3, characterized in that Generating the three-dimensional image of the newly added operation object according to the M initial two-dimensional images includes: Performing stretching processing on each initial two-dimensional image respectively to obtain a three-dimensional image of each initial two-dimensional image; Acquire feature points in each three-dimensional image to obtain M initial feature point sets, and filter the feature points in each initial feature point set to obtain M target feature point sets; Projecting the feature points in each target feature point set onto a two-dimensional plane to obtain a target two-dimensional image corresponding to each initial two-dimensional image; Input each target two-dimensional image into a three-dimensional image generation network to obtain a three-dimensional image corresponding to each target two-dimensional image; The M three-dimensional images are synthesized according to the viewing angle information to obtain a three-dimensional image of the newly added operation object.

5. The method according to claim 3, characterized in that: Generating the remote guidance image according to the three-dimensional image and the working environment image comprises: generating image data of the operation action in the working environment image according to the collaboration content and the three-dimensional image, and generating operation information corresponding to the operation action; The working environment image to which the image data of the operation information and the operation action are added is determined as the remote guidance image.

6. The method according to claim 1, characterized in that The working environment image includes a person image. After acquiring the working environment image sent by the on-site end, the method further includes: Identifying a working area in the working environment image; Determining whether the human image is located in the operation area; When the person image is located in the working area, determining that there is no abnormality in the working environment image; When the person image is not located in the working area, it is determined that there is an abnormality in the working environment image, and prompt information is generated according to the position information of the person image in the working environment image, and the prompt information is sent to the on-site end.

7. The method according to claim 1, characterized in that Sending the remote guidance image to the on-site end includes: The remote guidance image is sent to a cloud server, and the remote guidance image is stored in the blockchain to which the remote end belongs through the cloud server, and the remote guidance image is sent to the on-site end through the cloud server, wherein the cloud server is also used to store the collaboration establishment request sent by the on-site end.

8. A remote collaborative interactive device, characterized in that: include: A receiving unit, configured to receive a collaboration establishment request sent by a field end, and acquire an operating environment image sent by the field end, wherein the collaboration establishment request is used to request the establishment of a remote collaboration task between the field end and a remote end; A generating unit, configured to receive the information to be coordinated sent by the on-site end, and generate a remote guidance image according to the information to be coordinated and the working environment image, wherein the remote guidance image includes image data of the operation action and image data of the operation object; A sending unit is used to send the remote guidance image to the on-site end, wherein the on-site end processes the operation object according to the image data of the operation action.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the remote collaborative interaction method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the remote collaborative interaction method described in any one of claims 1 to 7.

Images