Data processing method and storage medium for linking online exhibitions with offline exhibitions

Abstract

The present application relates to a data processing method and storage medium for the linkage between an online exhibition and an offline exhibition, wherein the data processing method for the linkage between an online exhibition and an offline exhibition renders a virtual exhibition scene through a server based on image data and audio data of online exhibitors collected by online exhibition participating equipment and real-time data of the online exhibition collected by offline exhibition participating equipment, and obtains virtual exhibition scene data and sends it to VR glasses in the online exhibition participating equipment, so that the virtual exhibition scene has both online exhibitors and offline exhibition scenes, and then offline exhibitors can watch the offline exhibition scenes presented in the VR glasses, and integrate into the offline exhibition in the form of "virtual people", so that the online exhibitors can interact with the offline exhibitors in real time with strong immersion.

Description

Technical Field

[0001] The present application relates to the technical field of online exhibitions, and in particular to a data processing method and storage medium for linking online exhibitions with offline exhibitions. Background Art

[0002] At present, the global mobility of people is restricted, and a large number of exhibitions are gradually developing in the form of online and offline synchronization. The traditional online exhibition and offline exhibition linkage solution generally adopts the online live broadcast method. Specifically, the live video stream of the offline exhibition is pushed to the remote client, so that users can also observe the development of the online exhibition and obtain exhibit information on the remote client.

[0003] However, this traditional online and offline exhibition linkage solution using online live broadcast still has the problem of weak interactivity and poor immersion between offline exhibitors and online exhibitors, which makes it difficult for offline exhibitors to integrate into the offline exhibition scene. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method and storage medium for the linkage between online and offline exhibitions to address the problem that the traditional online and offline exhibition linkage solution has weak interactivity and poor immersion between offline exhibitors, which makes it difficult for offline exhibitors to integrate into the offline exhibition scene.

[0005] On the one hand, the present application provides a data processing method for linking an online exhibition with an offline exhibition, which is applied to an exhibition data linkage system, wherein the exhibition data linkage system includes an online exhibition participating device, a server, and multiple offline exhibition participating devices, and the exhibition data linkage system includes an offline exhibition participating device, a server, and multiple online exhibition participating devices. The data processing method for linking an online exhibition with an offline exhibition includes:

[0006] The online exhibition participating device receives the joining request through the VR glasses in the online exhibition participating device;

[0007] The online data collection device in the online exhibition participating device collects the image data and audio data of the online exhibitors and sends them to the server;

[0008] The offline exhibition participating device receives the joining request through the processing terminal in the offline exhibition participating device, collects the offline exhibition real-time data through the offline data collecting device in the offline exhibition participating device and sends it to the server;

[0009] The server renders the virtual exhibition scene based on the image data and audio data of the online exhibitors and the real-time data of the offline exhibition to obtain the virtual exhibition scene data, so that the virtual exhibition scene has both the online exhibitors and the offline exhibition scene;

[0010] The server sends the virtual exhibition scene data to the VR glasses in the online exhibition participating devices.

[0011] Furthermore, the online data collection device in the online exhibition participation device collects image data and audio data of online exhibitors and sends them to the server, including:

[0012] The depth camera in the online data acquisition device collects the online exhibition images of online exhibitors in real time and sends them to the server.

[0013] Furthermore, the server renders the virtual exhibition scene according to the image data and audio data of the online exhibitors and the real-time data of the offline exhibition to obtain the virtual exhibition scene data, so that the virtual exhibition scene has both the online exhibitors and the offline exhibition scene, including:

[0014] The server obtains the online exhibition image and audio data of each online exhibitor in real time;

[0015] The server performs image matting processing on the online exhibition image of each online exhibitor to obtain full-body portrait data of each online exhibitor.

[0016] Furthermore, the server renders the virtual exhibition scene according to the image data and audio data of the online exhibitors and the real-time data of the offline exhibition to obtain the virtual exhibition scene data, so that the virtual exhibition scene has both the online exhibitors and the offline exhibition scene, and further includes:

[0017] The server obtains real-time data of offline exhibitions and uses 3D modeling technology to construct virtual exhibition scenes;

[0018] The server adds the full-body portrait data of each online exhibitor to the virtual exhibition scene in the form of a sticker;

[0019] The server performs 3D rendering based on the mapped virtual exhibition scene to obtain a 3D rendered virtual exhibition scene.

[0020] Furthermore, after the offline exhibition participating device receives a membership request through a processing terminal in the offline exhibition participating device, and collects offline exhibition real-time data through an offline data collection device in the offline exhibition participating device and sends the data to the server, the method further includes:

[0021] The server sends the full-body portrait data of each online exhibitor to the laser projector of the offline exhibition participant equipment;

[0022] The laser projector projects the full-body portrait data of each online exhibitor onto the holographic screen of the offline exhibition equipment.

[0023] Furthermore, after the offline exhibition participating device receives the joining request through the processing terminal in the offline exhibition participating device, and collects offline exhibition real-time data through the offline data collecting device in the offline exhibition participating device and sends it to the server, the method further includes:

[0024] The server sends the audio data of each online exhibitor to the voice playback device of the offline exhibition participant device;

[0025] The language playback device analyzes and plays the audio data of each online exhibitor.

[0026] Furthermore, the offline exhibition participation device receives a membership request through a processing terminal in the offline exhibition participation device, collects offline exhibition real-time data through an offline data collection device in the offline exhibition participation device and sends the data to the server, including:

[0027] The offline data collection device in the offline exhibition participating device collects the audio data of the offline exhibition venue, and sends the audio data of the offline exhibition venue to the server;

[0028] The server sends the audio data of the offline exhibition venue to the VR glasses in the online exhibition participating devices.

[0029] Furthermore, the method further comprises:

[0030] The online exhibition participating device sends a query instruction to the server;

[0031] The server receives the query instruction, retrieves the historical exhibition videos and / or exhibit information stored in the server, and sends them to the VR glasses in the online exhibition participating devices.

[0032] On the other hand, the present application also provides an exhibition data linkage system, which applies the data processing method for linking online exhibitions with offline exhibitions as mentioned above, including:

[0033] Offline exhibition participation equipment;

[0034] Server, which communicates with offline exhibition participating devices;

[0035] A plurality of online exhibition participating devices, each of which is in communication connection with the server.

[0036] On the other hand, the present application also provides a computer storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the data processing method for linking online exhibitions with offline exhibitions as mentioned in the above content is implemented.

[0037] The present application also provides a data processing method and storage medium for the linkage between online exhibitions and offline exhibitions. The server renders a virtual exhibition scene based on the image data and audio data of online exhibitors collected by online exhibition participating devices and the real-time data of the online exhibition collected by offline exhibition participating devices. The virtual exhibition scene data is sent to the VR glasses in the online exhibition participating devices, so that the virtual exhibition scene has both online exhibitors and offline exhibition scenes. Then, offline exhibitors can watch the offline exhibition scenes presented in the VR glasses, and integrate into the offline exhibition in the form of "virtual people", so that the online exhibitors can interact with the offline exhibitors in real time with strong immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a data processing method for linking online exhibitions with offline exhibitions provided in one embodiment of the present application.

[0039] Figure 2 A schematic diagram of the structure of an exhibition data linkage system provided in one embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] This application provides a data processing method for linking online exhibitions with offline exhibitions. It should be noted that the data processing method for linking exhibitions with offline exhibitions provided in this application is applicable to any type of exhibition.

[0042] In addition, the data processing method for linking online exhibitions with offline exhibitions provided in this application does not limit its execution subject. Optionally, the execution subject of the data processing method for linking online exhibitions with offline exhibitions provided in this application can be an exhibition data linkage system.

[0043] like Figure 1 As shown, in one embodiment of the present application, the data processing method for linking the online exhibition with the offline exhibition includes:

[0044] S100, the online exhibition participating device receives a joining request through the VR glasses in the online exhibition participating device.

[0045] Specifically, VR glasses can block people's vision and hearing to the outside world, and guide online participants to feel like they are in a virtual environment. The display principle is that the left and right eye screens of VR glasses display the images of the left and right eyes respectively, and the human eye obtains this different information and creates a three-dimensional sense in the mind.

[0046] At the offline exhibition site, after debugging the equipment, the management personnel send a signal containing a joining request to the online exhibition participating equipment and the offline exhibition participating equipment simultaneously or successively through the remote server. The server can be directly set up at the offline exhibition site or not.

[0047] The online exhibition participation devices include VR glasses, whose internal communication modules can receive signals containing membership requests.

[0048] S200, the online data collection device in the online exhibition participating device collects the image data and audio data of the online exhibitors and sends them to the server.

[0049] Specifically, the online exhibition participation device also includes an online camera set up facing the online exhibitors, and the online camera captures and collects image data of offline exhibitors. The online camera can communicate with the VR glasses through Bluetooth or other means to send the image data of the online exhibitors to the VR glasses.

[0050] The online exhibition participation equipment also includes an offline audio device, which can collect audio data output by online exhibitors. The online audio device can communicate with the VR glasses through Bluetooth or other methods to send the image data of the online exhibitors to the VR glasses.

[0051] The VR glasses send the image and audio data of online exhibitors to the server.

[0052] S300, the offline exhibition participating device receives a joining request through a processing terminal in the offline exhibition participating device, collects offline exhibition real-time data through an offline data collection device in the offline exhibition participating device and sends it to a server.

[0053] Specifically, the offline exhibition participation equipment may include multiple offline cameras evenly distributed at the offline exhibition site. To enhance the interactive experience, the offline cameras may be cameras that can automatically adjust the shooting angle, replace the eyes of online exhibitors, and collect the images "seen" by online exhibitors. These images are integrated into real-time offline exhibition data in the form of images.

[0054] Online exhibitors are those who participate in the exhibition at home or in other remote locations. They cannot attend the offline exhibition in person and participate in the exhibition remotely through VR glasses. Offline exhibitors are those who are at the offline exhibition site and can actually see the exhibits.

[0055] S400, the server renders a virtual exhibition scene based on the image data and audio data of the online exhibitors and the real-time data of the offline exhibition to obtain virtual exhibition scene data, so that the virtual exhibition scene has both the online exhibitors and the offline exhibition scene.

[0056] Specifically, both the online camera and the offline camera can be depth cameras with depth information. The image in the offline exhibition real-time data is used as the scene map, and the image data of the online exhibitors is used as the insertion map. The scene map and the insertion map are superimposed and rendered to obtain the virtual exhibition scene data.

[0057] S500, the server sends the virtual exhibition scene data to the VR glasses in the online exhibition participating devices.

[0058] Specifically, this method allows remote online exhibitors to participate in offline exhibition scenes and interact with offline exhibitors, exhibits, etc.

[0059] In this embodiment, the server renders a virtual exhibition scene based on the image data and audio data of online exhibitors collected by the online exhibition participating equipment and the real-time data of the online exhibition collected by the offline exhibition participating equipment, and the obtained virtual exhibition scene data is sent to the VR glasses in the online exhibition participating equipment, so that the virtual exhibition scene has both online exhibitors and offline exhibition scenes, and then the offline exhibitors can watch the offline exhibition scenes presented in the VR glasses, and integrate into the offline exhibition in the form of "virtual people", so that the online exhibitors can interact with the offline exhibitors in real time with strong immersion.

[0060] In one embodiment of the present application, S200 includes:

[0061] S210, the depth camera in the online data collection device collects online exhibition images of online exhibitors in real time and sends them to the server.

[0062] Specifically, the online exhibition images stimulated by the depth camera have depth information, which facilitates the subsequent superposition of the online exhibition images with images in the real-time data of the offline exhibition.

[0063] Optionally, the online data collection device also includes a fill light and a tripod. The tripod is used to fix and clamp the depth camera. There can be at least two fill lights with adjustable brightness and color tone, which are used to fill light for the shooting environment of the depth camera, so as to ensure that the light on the bodies of all the "virtual" portraits of the participants photographed by the depth camera is uniform and the light source is unified.

[0064] In one embodiment of the present application, S400 includes:

[0065] S410, the server obtains the online exhibition image and audio data of each online exhibitor in real time.

[0066] S420: The server performs image matting processing on the online exhibition image of each online exhibitor to obtain full-body portrait data of each online exhibitor.

[0067] Specifically, the S420 matting process can separate the full-body portrait and environmental background elements in the online exhibition image, shielding the complex background in which the online exhibitors are located.

[0068] The server deploys a real-time keying server, and the real-time keying module is located inside the server's embedded image processor. Different from traditional keying processing, the keying processing in S420 does not require the arrangement of green screen / blue screen, or asking users to provide triple map. Instead, it trains the deep neural network model through a large amount of portrait data and exclusively collected meeting-related scene data. In this way, after the online exhibition image of each online exhibitor is input into the deep neural network model, the deep neural network model can realize fully automatic real-time fine keying.

[0069] When the server is performing the matting processing, it controls the full-body portrait data of each online exhibitor to be displayed in a 1:1 full-body manner, which optimizes the on-site display effect of the online exhibitors and improves the realism of the virtual exhibition scene.

[0070] When the server is performing the image cutout processing, it also ensures that the full-body portrait of each online exhibitor is of the same size, so as to achieve a portrait display effect consistent with that of other offline exhibitors from a visual display level.

[0071] In one embodiment of the present application, S400 includes:

[0072] S430, the server obtains real-time data of the offline exhibition in real time and uses 3D modeling technology to construct a virtual exhibition scene.

[0073] S440, the server adds the full-body portrait data of each online exhibitor to the virtual exhibition scene in the form of a sticker.

[0074] S450, the server performs 3D rendering according to the mapped virtual exhibition scene to obtain a 3D rendered virtual exhibition scene.

[0075] Specifically, S430 can use modeling software such as 3Dmax to construct a virtual exhibition scene based on image data in real-time data of offline exhibitions.

[0076] S440 and S450 are performed synchronously. The images in the real-time data of the offline exhibition are used as scene maps, and the image data of online exhibitors are used as insert maps. The scene maps and insert maps are superimposed on each other for 3D rendering to obtain a virtual exhibition scene after 3D rendering.

[0077] In one embodiment of the present application, after S300, the method further includes:

[0078] S610, the server sends the full-body portrait data of each online exhibitor to the laser projector of the offline exhibition participating device.

[0079] S620, the laser projector projects the full-body portrait data of each online exhibitor onto the holographic screen of the offline exhibition participating equipment.

[0080] Specifically, the aforementioned embodiments explain how to present the offline exhibition scene in the VR glasses of online exhibitors, and in the present application, the status of online exhibitors also needs to be broadcast to the offline exhibition scene in real time, so that offline exhibition personnel, especially managers, can know which online exhibitors have participated in the exhibition and the status of these online exhibition personnel.

[0081] Specifically, multiple holographic projection frames are arranged on the holographic screen of the offline exhibition participating equipment, and the laser projector projects the full-body portrait data of each online exhibitor into a holographic projection frame, so that the actual offline exhibition personnel in the offline exhibition scene can watch the images and movements of the remote online exhibitors in real time.

[0082] In this embodiment, holographic projection is applied to the exhibition scene instead of the recorded video, which is more realistic. The holographic technology is used to enhance the communication experience. Compared with the traditional conference system, it can display the participants more flexibly, provide rich display forms such as full body standing, sitting, single person, and multiple people, and is not limited to the conference system interface.

[0083] In one embodiment of the present application, after S300, the data processing method for linking the online exhibition with the offline exhibition further includes:

[0084] S710, the server sends the audio data of each online exhibitor to the voice playback device of the offline exhibition participant device.

[0085] S720: The language playback device analyzes and plays the audio data of each online exhibitor.

[0086] Specifically, by executing S710 to S720, actual offline exhibition personnel in the offline exhibition scene can not only watch the images and actions of each remote online exhibitor in real time, but also hear the voice of each online exhibitor, achieving the effect of audio and video synchronization.

[0087] In one embodiment of the present application, the S300 includes:

[0088] S310, the offline data collection device in the offline exhibition participating device collects the audio data of the offline exhibition venue, and sends the audio data of the offline exhibition venue to the server.

[0089] S320, the server sends the audio data of the offline exhibition venue to the VR glasses in the online exhibition participating devices.

[0090] Specifically, by executing S310 to S320, each remote online exhibitor can not only watch the pictures, exhibits, and actions of each offline exhibitor in real time, but also hear the voice of each offline exhibitor, achieving the effect of audio and video synchronization.

[0091] In one embodiment of the present application, the data processing method for linking the online exhibition with the offline exhibition also includes:

[0092] S810, the online exhibition participating device sends a query instruction to the server.

[0093] S820, the server receives the query instruction, retrieves the historical exhibition videos and / or exhibit information stored in the server, and sends them to the VR glasses in the online exhibition participating device.

[0094] Specifically, in this embodiment, the online exhibition participating device sends a review instruction to the server. The server receives the review instruction, retrieves the historical exhibition videos and / or exhibit information stored in the server, and sends it to the VR glasses in the online exhibition participating device, so that online exhibitors can review the historical exhibition videos before the exhibition starts, browse the exhibit information of historical exhibits through the UI interface, and preview the exhibit information of the exhibits participating in this exhibition.

[0095] During the exhibition, the offline data collection equipment of the offline exhibition participating equipment can record the exhibition video of this exhibition and use the exhibition video as a historical exhibition video.

[0096] After the exhibition, online participants can watch the video of the exhibition.

[0097] The application also provides an exhibition data linkage system.

[0098] In one embodiment of the present application, the exhibition data linkage system applies the data processing method for linking online exhibitions with offline exhibitions as mentioned in the above embodiments.

[0099] like Figure 2 As shown, the exhibition data linkage system includes an offline exhibition participation device 100, a server 200 and a plurality of online exhibition participation devices 300. The server 200 is connected in communication with the offline exhibition participation device 100. Each online exhibition participation device 300 is connected in communication with the server 200.

[0100] Specifically, it should be noted that, for the sake of brevity, the offline exhibition participating device 100, the server 200 and the multiple online exhibition participating devices 300 are only numbered in this embodiment, and are not numbered in the embodiment of the aforementioned data processing method for linking multiple online exhibitions with offline exhibitions.

[0101] The present application also provides a computer storage medium.

[0102] In one embodiment of the present application, a computer storage medium stores a computer program, characterized in that when the program is executed by a processor, the data processing method for linking online exhibitions with offline exhibitions as mentioned in the above-mentioned embodiment is implemented.

[0103] The technical features of the above-described embodiments may be arbitrarily combined, and the execution order of the method steps is not limited. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method for linking online exhibitions with offline exhibitions, characterized in that: Applied to an exhibition data linkage system, the exhibition data linkage system includes an offline exhibition participating device, a server and multiple online exhibition participating devices, and the data processing method for linking the online exhibition with the offline exhibition includes: The online exhibition participating device receives the joining request through the VR glasses in the online exhibition participating device; The online data collection equipment in the online exhibition participation equipment collects the image data and audio data of the online exhibitors and sends them to the server; the offline exhibition participation equipment includes multiple offline cameras evenly distributed at the offline exhibition site. To enhance the interactive experience, the offline cameras are cameras that can automatically adjust the shooting angle, replacing the eyes of the online exhibitors to collect the images seen by the online exhibitors. These images are integrated into real-time data of the offline exhibition in the form of images; The offline exhibition participating device receives the joining request through the processing terminal in the offline exhibition participating device, collects the offline exhibition real-time data through the offline data collecting device in the offline exhibition participating device and sends it to the server; The server renders the virtual exhibition scene based on the image data and audio data of the online exhibitors and the real-time data of the offline exhibition to obtain the virtual exhibition scene data, so that the virtual exhibition scene has both the online exhibitors and the offline exhibition scene; The server sends the virtual exhibition scene data to the VR glasses in the online exhibition participating devices; After the offline exhibition participating device receives the joining request through the processing terminal in the offline exhibition participating device, and collects offline exhibition real-time data through the offline data collecting device in the offline exhibition participating device and sends it to the server, the method further includes: The server sends the full-body portrait data of each online exhibitor to the laser projector of the offline exhibition participating equipment; The laser projector projects the full-body portrait data of each online exhibitor onto the holographic screen of the offline exhibition equipment.

2. The data processing method for linking online exhibitions with offline exhibitions according to claim 1 is characterized in that: The online data collection device in the online exhibition participation device collects image data and audio data of online exhibitors and sends them to the server, including: The depth camera in the online data acquisition device collects the online exhibition images of online exhibitors in real time and sends them to the server.

3. The data processing method for linking online exhibitions with offline exhibitions according to claim 2 is characterized in that: The server renders the virtual exhibition scene according to the image data and audio data of the online exhibitors and the real-time data of the offline exhibition to obtain the virtual exhibition scene data, so that the virtual exhibition scene has both the online exhibitors and the offline exhibition scene, including: The server obtains the online exhibition image and audio data of each online exhibitor in real time; The server performs image matting processing on the online exhibition image of each online exhibitor to obtain full-body portrait data of each online exhibitor.

4. The data processing method for linking online exhibitions with offline exhibitions according to claim 3 is characterized in that: The server renders the virtual exhibition scene according to the image data and audio data of the online exhibitors and the real-time data of the offline exhibition to obtain the virtual exhibition scene data, so that the virtual exhibition scene has both the online exhibitors and the offline exhibition scene, and also includes: The server obtains real-time data of offline exhibitions and uses 3D modeling technology to construct virtual exhibition scenes; The server adds the full-body portrait data of each online exhibitor to the virtual exhibition scene in the form of a sticker; The server performs 3D rendering based on the mapped virtual exhibition scene to obtain a 3D rendered virtual exhibition scene.

5. The data processing method for linking online exhibitions with offline exhibitions according to claim 4 is characterized in that: After the offline exhibition participating device receives the joining request through the processing terminal in the offline exhibition participating device, and collects offline exhibition real-time data through the offline data collecting device in the offline exhibition participating device and sends it to the server, the method further includes: The server sends the audio data of each online exhibitor to the voice playback device of the offline exhibition participant device; The language playback device analyzes and plays the audio data of each online exhibitor.

6. The data processing method for linking online exhibitions with offline exhibitions according to claim 5 is characterized in that: The offline exhibition participating device receives a joining request through a processing terminal in the offline exhibition participating device, collects offline exhibition real-time data through an offline data collection device in the offline exhibition participating device and sends the data to a server, including: The offline data collection device in the offline exhibition participating device collects the audio data of the offline exhibition venue, and sends the audio data of the offline exhibition venue to the server; The server sends the audio data of the offline exhibition venue to the VR glasses in the online exhibition participating devices.

7. The data processing method for linking online exhibitions with offline exhibitions according to claim 6 is characterized in that: The method further comprises: The online exhibition participating device sends a query instruction to the server; The server receives the query instruction, retrieves the historical exhibition videos and / or exhibit information stored in the server, and sends them to the VR glasses in the online exhibition participating devices.

8. An exhibition data linkage system, applying the data processing method for linking online exhibitions with offline exhibitions as described in any one of claims 1 to 7, characterized in that: include: Offline exhibition participation equipment; Server, which communicates with offline exhibition participating devices; A plurality of online exhibition participating devices, each of which is in communication connection with the server.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method for linking an online exhibition with an offline exhibition as described in any one of claims 1 to 7 is implemented.

Images