XR information transmission control method, device, system and storage medium

Abstract

The present invention discloses an XR information transmission control method, device, system and storage medium, the method comprising: controlling a cloud server to parse and virtually render received XR information to generate a virtual 3D image; the XR information is transmitted by an XR application; the XR information comprises: user posture information and scene information; compressing the virtual 3D image and the original audio signal frame by frame respectively to generate image compression data and audio compression data, and transmitting the image compression data and the audio compression data as compressed data to an edge end; controlling the edge end to decompress and detect changes in the compressed data to generate a detection result, and sending the detection result to a terminal; the detection result comprises the changed virtual 3D image or audio information and the change type thereof; controlling the terminal to update the corresponding data in the client according to the detection result, so as to reduce the data transmission volume and bandwidth occupancy, thereby improving the efficiency and stability of XR information transmission.

Description

Technical Field

[0001] The present invention relates to the field of extended reality technology, and in particular to an XR information transmission control method, device, system and storage medium. Background Art

[0002] Extended reality (XR) technology includes mixed reality (MR), augmented reality (AR), virtual reality (VR) and any other technology that integrates reality and virtuality. At present, there are two main types of XR applications: one is that the entire XR task is processed by the terminal device, and the other is that the cloud server processes the XR task and then streams the processed data such as 3D models, images and audio to the terminal device.

[0003] Existing terminal devices responsible for processing XR applications have difficulty meeting the requirements of computing and storage resources. In cloud-based XR technology, terminal devices need to have strong communication capabilities with the cloud, and they also need to stream processed 3D models, images, and audio data to the terminal devices, which takes up a lot of network bandwidth resources. After the XR application is processed in the cloud, there is a certain delay in transmitting the data to the terminal device. Coupled with network fluctuations, there is a large delay in transmitting XR data to the terminal, further reducing the user experience. Therefore, the existing XR technology has high hardware requirements for terminal devices, large XR data transmission volume and bandwidth occupancy, poor information processing efficiency, and low stability of XR services. Summary of the invention

[0004] The present invention provides an XR information transmission control method, device, system and storage medium to reduce the data transmission volume and bandwidth occupancy, thereby improving the efficiency and stability of XR information transmission.

[0005] The present invention provides an XR information transmission control method, comprising: controlling a cloud server to parse and virtually render received first XR information to generate a virtual 3D image; the first XR information is transmitted by an XR application; the first XR information includes: user posture information and scene information; compressing the virtual 3D image and the original audio signal frame by frame respectively to generate image compression data and audio compression data, and transmitting the image compression data and the audio compression data as compressed data to an edge end;

[0006] Controlling the edge end to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type;

[0007] The control terminal updates the corresponding data in the client according to the detection result.

[0008] As a preferred solution, the present invention pre-processes and compresses the 3D models, images, audio and other data required for XR applications under the architecture of cloud servers, edge terminals and terminals, generates corresponding data feature representations, and then sends the data to the edge terminal, thereby reducing the hardware cost of the terminal device and reducing the amount of data transmission, thereby reducing the occupancy of network bandwidth resources; after the XR data reaches the edge terminal, by using change detection technology, the changed part of the XR data is found, and the updated data is sent to the client, thereby completing the transmission and update of the XR data. Since only the changed data is transmitted, the transmission delay is reduced, and the data transmission amount and bandwidth occupancy are further reduced, thereby improving the efficiency and stability of XR information transmission.

[0009] Furthermore, the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, which specifically includes:

[0010] Convert the virtual 3D image into optical information, and obtain the light information in each direction of each pixel to generate a number of light field information; wherein each pixel corresponds to a light field information;

[0011] Each piece of light field information is sampled, each piece of light field information is divided into a plurality of small blocks, each of the small blocks is encoded and compressed, and image compression data is generated.

[0012] Further, the edge end is controlled to decompress and detect changes in the compressed data to generate a detection result, specifically including:

[0013] Decompressing the image compressed data based on the light field to obtain first 3D data;

[0014] After denoising and downsampling the second 3D data of two preset time points of the same scene, the two second 3D data are registered; feature extraction is performed on the two registered second 3D data respectively to obtain corresponding 3D data features; the 3D data features of the two second 3D data are compared to detect first change data of the two second 3D data; the first change data is analyzed to obtain an image change type; and the first change data and the corresponding image change type are used as a first detection result.

[0015] Furthermore, the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, which specifically includes:

[0016] The original audio signal is sampled and filtered to remove interference signals; the sampled and filtered audio signal is divided into frames, and the spectrum analysis is performed on each frame of the audio signal to extract the frequency and amplitude information; the frequency and amplitude information are encoded, packaged and compressed using CELT and SILK technology to generate audio compression data.

[0017] Further, the edge end is controlled to decompress and detect changes in the compressed data to generate a detection result, specifically including:

[0018] Decompressing the audio compressed data in a corresponding manner to obtain a first audio signal; preprocessing the first audio signal, wherein the preprocessing includes denoising and reducing interference;

[0019] Perform feature extraction on the preprocessed first audio signal to obtain audio data features; match the audio data features of the second audio signals at two preset time points in the same scene to detect second change data of the second audio signals at the two preset time points; analyze the second change data to obtain an audio change type; and use the second change data and the corresponding audio change type as a second detection result.

[0020] As a preferred solution, the present application pre-processes and compresses the virtual 3D image and the original audio signal on the cloud server, generates corresponding data feature representations, namely, image compression data and audio compression data, and sends them to the edge end, thereby reducing the amount of data transmission and thus reducing the occupancy of network bandwidth resources. The edge end decompresses and detects changes in the image compression data and audio compression data, respectively, finds the changed parts of the XR data, and sends the updated data to the client, thereby completing the transmission and update of the XR data, further reducing the amount of data transmission and bandwidth occupancy. Change detection technology can quickly detect changes in the transmitted data, thereby quickly and timely transmitting the corresponding data, reducing the delay in data transmission. At the same time, change detection technology only transmits the changed part of the data, which improves the transmission speed and response speed, thereby improving the efficiency and stability of XR services.

[0021] Furthermore, the control terminal updates the corresponding data in the client according to the detection result, specifically:

[0022] According to the image change type of the first detection result, the first change data is transmitted to the client for data updating; according to the audio change type of the second detection result, the second change data is transmitted to the client for data updating.

[0023] As a preferred solution, after performing change detection on the transmitted XR data at the edge, the present application obtains the first change data of the image and the second change data of the audio and transmits them to the client for data update. Only the changed part of the data is transmitted, thus avoiding the transmission of a large amount of redundant data, effectively reducing the data transmission volume, and saving bandwidth resources.

[0024] Accordingly, the present invention also provides an XR information transmission control device, comprising: a cloud server control module, an edge control module and a terminal control module;

[0025] The cloud server control module is used to control the cloud server to parse and virtually render the received first XR information to generate a virtual 3D image; the first XR information is transmitted by the XR application; the first XR information includes: user posture information and scene information; the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, and the image compression data and the audio compression data are transmitted to the edge end as compressed data;

[0026] The edge control module is used to control the edge to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type;

[0027] The terminal control module is used to control the terminal to update the corresponding data in the client according to the detection result.

[0028] As a preferred solution, the device of the present invention pre-processes and compresses the 3D model, image, audio and other data required for the XR application through the cloud server control module under the architecture of the cloud server, edge terminal and terminal, generates the corresponding data feature representation, and then sends the data to the edge terminal, reducing the hardware cost of the terminal device, and reducing the amount of data transmission, thereby reducing the occupation of network bandwidth resources; after the XR data arrives at the edge terminal, the edge terminal control module uses the change detection technology to find the changed part of the XR data, and sends the updated data to the client, and the terminal control module completes the transmission and update of the XR data. Since only the changed data is transmitted, the transmission delay is reduced, and the data transmission amount and bandwidth occupation are further reduced, thereby improving the efficiency and stability of XR information transmission.

[0029] Accordingly, the present invention also provides an XR information transmission control system, comprising: a cloud server, an edge terminal, a terminal and a control terminal;

[0030] The cloud server is used to receive the first XR information; parse and virtually render the received first XR information to generate a virtual 3D image; the first XR information is transmitted by the XR application; the first XR information includes: user posture information and scene information; compress the virtual 3D image and the original audio signal frame by frame to generate image compression data and audio compression data, and use the image compression data and audio compression data as compressed data; and transmit the compressed data to the edge end;

[0031] The edge end is used to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type;

[0032] The terminal is used to update the corresponding data in the client according to the detection result;

[0033] The control end is used to control the cloud server, edge end and terminal to perform the above corresponding operations.

[0034] Accordingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute an XR information transmission control method as described in the content of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of an embodiment of the XR information transmission control method provided by the present invention;

[0036] Figure 2 It is a structural schematic diagram of an embodiment of the XR information transmission control system provided by the present invention;

[0037] Figure 3 It is a structural schematic diagram of an embodiment of the XR information transmission control device provided by the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Embodiment 1

[0040] Please refer to Figure 1, an XR information transmission control method provided by an embodiment of the present invention is applied to a control end of an XR information transmission control system, the control system comprising: a cloud server, an edge end, a terminal and a control end;

[0041] The XR information transmission control method comprises steps S101-S103:

[0042] Step S101: Control the cloud server to parse and virtually render the received first XR information to generate a virtual 3D image; the first XR information is transmitted by the XR application; the first XR information includes: user posture information and scene information; compress the virtual 3D image and the original audio signal frame by frame to generate image compression data and audio compression data, and transmit the image compression data and audio compression data to the edge end as compressed data;

[0043] Furthermore, the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, which specifically includes:

[0044] Convert the virtual 3D image into optical information, and obtain the light information in each direction of each pixel to generate a number of light field information; wherein each pixel corresponds to a light field information;

[0045] Each piece of light field information is sampled, each piece of light field information is divided into a plurality of small blocks, each of the small blocks is encoded and compressed, and image compression data is generated.

[0046] Furthermore, the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, which specifically includes:

[0047] The original audio signal is sampled and filtered to remove interference signals; the sampled and filtered audio signal is divided into frames, and the spectrum analysis is performed on each frame of the audio signal to extract the frequency and amplitude information; the frequency and amplitude information are encoded, packaged and compressed using CELT and SILK technology to generate audio compression data.

[0048] In this embodiment, the first XR information transmitted by the XR application and received by the cloud server includes: user posture information and scene information; illustratively, the user posture information includes posture information such as the user's head orientation and hand movements; the scene information includes the light and texture of the environment, etc.; based on the received first XR information, virtual rendering is performed on the cloud server, and a virtual 3D image is generated. As a preferred embodiment, the virtual 3D image is a virtual 3D image from the user's perspective.

[0049] In this embodiment, for the compression of the virtual 3D image, a compression technology based on light field is used, that is, a method similar to photography is used to convert the 3D scene into optical information, and the light information in each direction of each pixel in the virtual 3D image is recorded. Each pixel obtains a light field, that is, a set of scene models (3D models) composed of multiple light rays. Each light field is sampled and divided into multiple small blocks, and each small block is encoded and compressed to complete the compression of the virtual 3D image.

[0050] In this embodiment, Opus encoding is used to compress the original audio signal, that is, the original audio signal is sampled and filtered to remove noise and other unnecessary signals. The audio signal is divided into frames, and each frame is subjected to spectrum analysis to extract frequency and amplitude information. The audio signal is encoded, packaged and compressed using CELT and SILK technology to generate Opus audio data.

[0051] Step S102: Control the edge end to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type;

[0052] Further, the edge end is controlled to decompress and detect changes in the compressed data to generate a detection result, specifically including:

[0053] Decompressing the image compressed data based on the light field to obtain first 3D data;

[0054] After denoising and downsampling the second 3D data of two preset time points of the same scene, the two second 3D data are registered; feature extraction is performed on the two registered second 3D data respectively to obtain corresponding 3D data features; the 3D data features of the two second 3D data are compared to detect first change data of the two second 3D data; the first change data is analyzed to obtain an image change type; and the first change data and the corresponding image change type are used as a first detection result.

[0055] Further, controlling the edge end to decompress and detect changes in the compressed data to generate a detection result specifically includes:

[0056] Decompressing the audio compressed data in a corresponding manner to obtain a first audio signal; preprocessing the first audio signal, wherein the preprocessing includes denoising and reducing interference;

[0057] Perform feature extraction on the preprocessed first audio signal to obtain audio data features; match the audio data features of the second audio signals at two preset time points in the same scene to detect second change data of the second audio signals at the two preset time points; analyze the second change data to obtain an audio change type; and use the second change data and the corresponding audio change type as a second detection result.

[0058] In this embodiment, the compressed data transmitted from the cloud is transmitted to the edge via WebRTC. After the edge receives the compressed data from the cloud, it decompresses it using corresponding technologies according to the data type, that is, the image compressed data uses light field-based decompression, and the audio compressed data uses Opus decompression.

[0059] In this embodiment, for change detection of image decompressed data, the obtained point cloud or 3D model, i.e., the first audio signal, is preprocessed. The preprocessing operations include: denoising, downsampling, and registration, etc., to ensure the accuracy of subsequent change detection. The first audio signals of two different time points of the same scene are registered, that is, they are aligned, and feature extraction is performed on the registered data. The features include normal vectors, curvatures, surface features, etc. By comparing the features of the three-dimensional data of the image at two time points, the changes in the image between the two time points, such as addition, deletion, and movement, are detected; the changed image data, i.e., the first change data, is saved, and each detected change is analyzed to determine the corresponding actual change type.

[0060] For change detection of audio decompressed data, the decompressed audio signal is preprocessed, and the preprocessing operations include: denoising and reducing sound interference, etc.; feature extraction is performed on the preprocessed audio data, and the features include short-time energy, zero-crossing rate and MFCC (Mel frequency cepstral coefficient), etc.

[0061] Match the audio data features of two different time points of the same scene, compare the similarity between the audio data features, and then detect the changes in the audio between the two time points; save the changed audio data, that is, the second change data, analyze the detected changes, and determine the corresponding actual change type, specifically:

[0062] The audio data of two different time points of the same scene are preprocessed, and the ICA algorithm is used to decompose the audio data into different independent components.

[0063] If the audio data is a mixed signal of several speakers, use the ICA algorithm to decompose the audio data into different independent components, each component corresponds to the sound signal of a speaker. Compare the decomposed components of the audio data at two different time points to obtain the changes in the audio, such as adding new sounds, deleting sounds, changes in sound intensity, etc.

[0064] For example, when the first speaker enters the scene to speak at a later time point, the sound signal of the first speaker will be decomposed by the ICA algorithm in the audio data at the later time point, that is, there will be one more independent component of the speaker. When the second speaker stops speaking or leaves the scene at a later time point, the sound signal of the second speaker will be separated by the ICA algorithm in the audio data at the later time point, that is, there will be one less independent component of the speaker.

[0065] The audio signal of the changed part is analyzed and processed to determine the corresponding actual change type, such as who is speaking and the content of the speech.

[0066] As one embodiment, an amplitude spectrum analysis is performed on the audio signal of the changed part to obtain the sound energy spectra of different speakers, and the sound intensity and speech content of the speakers are calculated according to the sound energy changes in the sound energy spectrum.

[0067] As one of the embodiments, the audio signal of the changed part is input into a speech recognition engine to generate corresponding text, thereby achieving automatic recognition and conversion into text.

[0068] Through the above analysis, it is possible to effectively process the changing part of the speech signal, and identify and analyze the voices and content of different speakers.

[0069] As one embodiment, sentiment analysis may be performed on the speech of different speakers to obtain information such as the speaker's emotional state.

[0070] Step S103: the control terminal updates the corresponding data in the client according to the detection result.

[0071] Furthermore, the control terminal updates the corresponding data in the client according to the detection result, specifically:

[0072] According to the image change type of the first detection result, the first change data is transmitted to the client for data updating; according to the audio change type of the second detection result, the second change data is transmitted to the client for data updating.

[0073] In this embodiment, the first change data is transmitted to the client for data update, specifically:

[0074] When the image change type of the virtual 3D image is addition or deletion, only the incremental image data is transmitted for data update instead of transmitting the entire new model. The method of transmitting the incremental part adopts the incremental update technology, that is, the virtual 3D image is divided into several fixed-size blocks, each block has a unique identifier, and when a block in the data is detected to have changed, only the changed part needs to be transmitted to the client without transmitting the entire image model, thereby reducing the amount of data that needs to be transmitted.

[0075] When the image change type of the virtual 3D image is displacement, rotation or scaling, etc., only the corresponding transformation matrix is ​​transmitted for data update, rather than retransmitting the entire model.

[0076] In this embodiment, the second change data is transmitted to the client for data update, specifically:

[0077] When the audio change type of the original audio signal is addition or deletion, only the newly added or deleted part of the audio signal is transmitted for data update, rather than retransmitting the entire audio file.

[0078] When the audio change type of the original audio signal is an attribute change, such as a change in sound intensity or timbre, only the corresponding change value is transmitted for data update, rather than retransmitting the entire audio file.

[0079] Therefore, the change detection phase will detect the changed virtual 3D images or audio data and update the client according to the type of change, which can reduce the amount of data transmission and bandwidth occupancy while ensuring the efficiency and stability of XR information transmission.

[0080] In this embodiment, please refer to Figure 2 It is a structural diagram of an XR information transmission control system, wherein the system comprises: a cloud server 201, an edge terminal 202, a terminal 203 and a control terminal 204;

[0081] The cloud server 201 is used to receive the first XR information; parse and virtually render the received first XR information to generate a virtual 3D image; the first XR information is transmitted by the XR application; the first XR information includes: user posture information and scene information; compress the virtual 3D image and the original audio signal frame by frame respectively to generate image compression data and audio compression data, and use the image compression data and audio compression data as compressed data; and transmit the compressed data to the edge end;

[0082] The edge end 202 is used to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type;

[0083] The terminal 203 is used to update the corresponding data in the client according to the detection result;

[0084] The control terminal 204 is used to control the cloud server, edge terminal and terminal to perform the above corresponding operations.

[0085] Implementing the embodiments of the present invention has the following effects:

[0086] The present invention, through the architecture of cloud server, edge terminal and terminal, pre-processes and compresses the 3D model, image, audio and other data required by XR application, generates the corresponding data feature representation, and then sends the data to the edge terminal, thereby reducing the hardware cost of the terminal device, and reducing the data transmission volume, thereby reducing the occupation of network bandwidth resources; after the XR data reaches the edge terminal, by using the change detection technology, the changed part of the XR data is found, and the updated data is sent to the client, thereby completing the transmission and update of the XR data. Since only the changed data is transmitted, the transmission delay is reduced, and the data transmission volume and bandwidth occupation are further reduced, thereby improving the efficiency and stability of XR information transmission.

[0087] Embodiment 2

[0088] Please refer to Figure 2 , an XR information transmission control device provided by an embodiment of the present invention, comprising: a cloud server control module 301, an edge terminal control module 302 and a terminal control module 303;

[0089] The cloud server control module 301 is used to control the cloud server to parse and virtually render the received first XR information to generate a virtual 3D image; the first XR information is transmitted by the XR application; the first XR information includes: user posture information and scene information; the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, and the image compression data and the audio compression data are transmitted to the edge end as compressed data;

[0090] The edge control module 302 is used to control the edge to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type;

[0091] The terminal control module 303 is used to control the terminal to update the corresponding data in the client according to the detection result.

[0092] The cloud server control module 301 includes: a first compression unit and a second compression unit;

[0093] The first compression unit is used to convert the virtual 3D image into optical information, and obtain the light information in each direction of each pixel to generate a plurality of light field information; wherein each pixel corresponds to a light field information; each of the light field information is sampled, each of the light field information is divided into a plurality of small blocks, each of the small blocks is encoded and compressed to generate image compression data.

[0094] The second compression unit is used to sample and filter the original audio signal to remove interference signals; divide the sampled and filtered audio signal into frames, perform spectrum analysis on each frame of the audio signal, and extract frequency and amplitude information; encode, package and compress the frequency and amplitude information using CELT and SILK technology to generate audio compression data.

[0095] The edge control module 302 includes: a first change detection unit and a second change detection unit;

[0096] The first change detection unit is used to decompress the image compression data based on the light field to obtain first 3D data;

[0097] After denoising and downsampling the second 3D data of two preset time points of the same scene, the two second 3D data are registered; feature extraction is performed on the two registered second 3D data respectively to obtain corresponding 3D data features; the 3D data features of the two second 3D data are compared to detect first change data of the two second 3D data; the first change data is analyzed to obtain an image change type; and the first change data and the corresponding image change type are used as a first detection result.

[0098] The second change detection unit is used to decompress the audio compression data corresponding to the encoding to obtain a first audio signal; preprocess the first audio signal, and the preprocessing includes denoising and reducing interference;

[0099] Perform feature extraction on the preprocessed first audio signal to obtain audio data features; match the audio data features of the second audio signals at two preset time points in the same scene to detect second change data of the second audio signals at the two preset time points; analyze the second change data to obtain an audio change type; and use the second change data and the corresponding audio change type as a second detection result.

[0100] The terminal control module 303 includes: an updating unit;

[0101] The updating unit is used to transmit the first change data to the client for data updating according to the image change type of the first detection result; and transmit the second change data to the client for data updating according to the audio change type of the second detection result.

[0102] The above-mentioned XR information transmission control device can implement the XR information transmission control method of the above-mentioned method embodiment. The options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of this application can refer to the contents of the above-mentioned method embodiment, and will not be repeated in this embodiment.

[0103] Implementing the embodiments of the present invention has the following effects:

[0104] The device of the present invention uses the architecture of cloud server, edge terminal and terminal, and pre-processes and compresses the 3D model, image, audio and other data required for XR application through the cloud server control module, generates the corresponding data feature representation, and then sends the data to the edge terminal, reducing the hardware cost of the terminal device, and reducing the data transmission volume, thereby reducing the occupation of network bandwidth resources; after the XR data arrives at the edge terminal, the edge terminal control module uses the change detection technology to find the changed part of the XR data, and sends the updated data to the client, and the terminal control module completes the transmission and update of the XR data. Since only the changed data is transmitted, the transmission delay is reduced, and the data transmission volume and bandwidth occupation are further reduced, thereby improving the efficiency and stability of XR information transmission.

[0105] Embodiment 3

[0106] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the XR information transmission control method described in any one of the above embodiments.

[0107] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0108] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0109] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0110] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0111] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0112] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for controlling XR information transmission, characterized in that: include: Controlling the cloud server to parse and virtually render the received first XR information to generate a virtual 3D image; The first XR information is transmitted by the XR application; The first XR information includes: user posture information and scene information; compressing the virtual 3D image and the original audio signal frame by frame respectively to generate image compression data and audio compression data, and transmitting the image compression data and audio compression data as compressed data to the edge end; wherein, compressing the virtual 3D image and the original audio signal frame by frame respectively to generate image compression data and audio compression data includes: converting the virtual 3D image into optical information, and obtaining the light information in each direction of each pixel to generate a plurality of light field information; wherein each pixel corresponds to a light field information; sampling each of the light field information, dividing each of the light field information into a plurality of small blocks, encoding and compressing each of the small blocks to generate image compression data; sampling and filtering the original audio signal to remove interference signals; dividing the sampled and filtered audio signal into frames, performing spectrum analysis on each frame of the audio signal, and extracting frequency and amplitude information; encoding, packaging and compressing the frequency and amplitude information using CELT and SILK technology to generate audio compression data; Controlling the edge end to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type; The control terminal updates the corresponding data in the client according to the detection result.

2. The XR information transmission control method according to claim 1, characterized in that: The controlling edge end decompresses and detects changes on the compressed data to generate a detection result, specifically including: Decompressing the image compressed data based on the light field to obtain first 3D data; After denoising and downsampling the second 3D data of two preset time points of the same scene, the two second 3D data are registered; feature extraction is performed on the two registered second 3D data respectively to obtain corresponding 3D data features; the 3D data features of the two second 3D data are compared to detect first change data of the two second 3D data; the first change data is analyzed to obtain an image change type; and the first change data and the corresponding image change type are used as a first detection result.

3. The XR information transmission control method according to claim 2, characterized in that: The controlling edge end decompresses and detects changes on the compressed data to generate a detection result, specifically including: Decompressing the audio compressed data in a corresponding manner to obtain a first audio signal; preprocessing the first audio signal, wherein the preprocessing includes denoising and reducing interference; Perform feature extraction on the preprocessed first audio signal to obtain audio data features; match the audio data features of the second audio signals at two preset time points in the same scene to detect second change data of the second audio signals at the two preset time points; analyze the second change data to obtain an audio change type; and use the second change data and the corresponding audio change type as a second detection result.

4. An XR information transmission control method according to any one of claims 1 to 3, characterized in that: The control terminal updates the corresponding data in the client according to the detection result, specifically: According to the image change type of the first detection result, the first change data is transmitted to the client for data updating; according to the audio change type of the second detection result, the second change data is transmitted to the client for data updating.

5. An XR information transmission control device, characterized in that: include: Cloud server control module, edge control module and terminal control module; The cloud server control module is used to control the cloud server to parse and virtually render the received first XR information to generate a virtual 3D image; the first XR information is transmitted by the XR application; the first XR information includes: user posture information and scene information; the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, and the image compression data and the audio compression data are transmitted to the edge end as compressed data; wherein the virtual 3D image and the original audio signal are compressed frame by frame to generate image compression data and audio compression data, including: Convert it into optical information, and obtain the light information in each pixel in each direction to generate a number of light field information; wherein each pixel corresponds to a light field information; sample each of the light field information, divide each of the light field information into a number of small blocks, encode and compress each of the small blocks to generate image compression data; sample and filter the original audio signal to remove interference signals; divide the sampled and filtered audio signal into frames, perform spectrum analysis on each frame of the audio signal, and extract frequency and amplitude information; encode, package and compress the frequency and amplitude information using CELT and SILK technology to generate audio compression data; The edge control module is used to control the edge to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type; The terminal control module is used to control the terminal to update the corresponding data in the client according to the detection result.

6. An XR information transmission control system, characterized in that: include: Cloud servers, edge terminals, terminals, and control terminals; The cloud server is used to receive the first XR information; parse and virtually render the received first XR information to generate a virtual 3D image; The first XR information is transmitted by the XR application; The first XR information includes: user posture information and scene information; compressing the virtual 3D image and the original audio signal frame by frame respectively to generate image compression data and audio compression data, and using the image compression data and audio compression data as compressed data; transmitting the compressed data to the edge end; wherein, compressing the virtual 3D image and the original audio signal frame by frame respectively to generate image compression data and audio compression data includes: converting the virtual 3D image into optical information, and obtaining light information in each direction in each pixel to generate a plurality of light field information; wherein each pixel corresponds to a light field information; sampling each of the light field information, dividing each of the light field information into a plurality of small blocks, encoding and compressing each of the small blocks to generate image compression data; sampling and filtering the original audio signal to remove interference signals; framing the sampled and filtered audio signal, performing spectrum analysis on each frame of the audio signal, and extracting frequency and amplitude information; encoding, packaging and compressing the frequency and amplitude information using CELT and SILK technology to generate audio compression data; The edge end is used to decompress and detect changes in the compressed data, generate a detection result, and send the detection result to the terminal; the detection result includes the changed virtual 3D image or audio information and the change type; The terminal is used to update the corresponding data in the client according to the detection result; The control end is used to control the cloud server, edge end and terminal to perform the above corresponding operations.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute an XR information transmission control method as described in any one of claims 1 to 4.

Images