Control method of virtual reality equipment, virtual reality equipment and storage medium
By embedding FPGA modules in VR devices for image stitching, the problem of high SOC computational overhead is solved, achieving lower power consumption and higher quality full-color perspective effects.
Patent Information
- Application Number
- CN202310827957.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-06
- Publication Date
- 2025-01-07
AI Technical Summary
When implementing full-color perspective functionality in existing VR devices, the SOC needs to process invalid frames, resulting in problems such as high computational overhead, slow user response, low real-time image rendering quality, and high power consumption.
An FPGA module is embedded between the camera module and the SOC. The FPGA module completes the line-by-line stitching of a single frame image and sends the stitched frame to the SOC, reducing the computational burden on the SOC.
It effectively releases the computing power of the SOC, reduces the image processing power consumption of virtual reality devices, and improves user response speed and image rendering quality while realizing full-color perspective function.
Smart Images
Figure CN119271161A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of virtual reality device control, and in particular to a control method for virtual reality device, virtual reality device and storage medium. Background Art
[0002] In the current VR (Virtual Reality) industry, when VR realizes the full-color perspective function, it is usually necessary to control the Camera data stream to be transmitted to the SOC (System on Chip) through the MIPI (Mobile Industry Processor Interface) interface. The SOC then receives a frame of image from the camera for image stitching processing.
[0003] However, this method will transfer invalid frames in the camera data stream that are not related to image stitching processing to the SOC for processing. When performing stitching, the SOC also needs to identify invalid frames from the camera data stream and remove them, which increases the computational overhead of the SOC and easily causes negative effects on VR devices, such as slower user action response, lower real-time image rendering quality, and increased power consumption. Therefore, a solution that does not rely on the SOC to control virtual reality devices is needed to reduce the computational overhead of the SOC.
[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The main purpose of the present invention is to provide a control method for a virtual reality device, aiming to solve the problem of how to reduce the calculation overhead of SOC.
[0006] To achieve the above object, the present invention provides a method for controlling a virtual reality device, the method comprising:
[0007] Determine the display content corresponding to the current pixel row according to the image frame data received by each of the input pins;
[0008] Determine whether the current pixel row is the last row corresponding to the current picture frame;
[0009] If yes, send the current picture frame to the SOC;
[0010] Otherwise, the next pixel row is used as the current pixel row, and the process jumps to executing the step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins.
[0011] Optionally, the step of determining whether the current pixel row is the last row corresponding to the current picture frame includes:
[0012] Determine whether the current pixel row is the last row according to the display position corresponding to the display content in the current picture frame; or
[0013] Determine whether the current pixel row is the last row according to the receiving order corresponding to the image frame data.
[0014] Optionally, the step of determining whether the current pixel row is the last row according to a display position corresponding to display content corresponding to the current pixel row in the current picture frame includes:
[0015] Determine whether the display position is at the bottom of the current picture frame, and if so, determine that the current pixel row is the last row; or,
[0016] Determine whether the splicing sequence corresponding to the display position is the last splicing sequence corresponding to the current picture frame, and if so, determine that the current pixel row is the last row.
[0017] Optionally, before the step of determining whether the current pixel row is the last row according to the receiving order corresponding to the image frame data, the step further includes:
[0018] Determine the receiving order according to the receiving time when each of the input pins receives the image frame data, wherein the image frame data with an earlier receiving time has an earlier corresponding receiving order;
[0019] The step of determining whether the current pixel row is the last row according to the receiving order corresponding to the image frame data comprises:
[0020] Determine whether the receiving order is the last receiving order corresponding to the image frame data, and if so, determine that the current pixel row is the last row.
[0021] Optionally, before the step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins, the step further includes:
[0022] Receive a long data packet sent by the Camera module to the input pin;
[0023] Unpacking the long data packet to obtain image frame data corresponding to the current pixel row, wherein the image frame data from different Camera modules have different field angles;
[0024] The step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins comprises:
[0025] Splicing the image frame data in order from left to right to obtain spliced image data corresponding to the current pixel row, wherein the field of view corresponding to the spliced image data is greater than the field of view corresponding to each of the image frame data;
[0026] The display content corresponding to the current pixel row is determined according to the spliced image data.
[0027] Optionally, the FPGA further includes a MIPI-RX core, and the step of unpacking the long data packet to obtain the image frame data corresponding to the current pixel row includes:
[0028] Calling the parser in the MIPI-RX core;
[0029] The long data packet is unpacked by the parser to obtain the image frame data.
[0030] Optionally, the FPGA includes an IP core, the output end includes a MIPI-CSI output line, and before the step of sending the current picture frame to the SOC, the step further includes:
[0031] Calling the wrapper in the IP core;
[0032] Encapsulating the current picture frame through the encapsulator;
[0033] The step of sending the current picture frame to the SOC comprises:
[0034] The packaged current picture frame is sent to the SOC via the MIPI-CSI output line.
[0035] Optionally, data is transmitted between the Camera module and the FPGA module based on the MIPI-CSI protocol.
[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a virtual reality device, which includes: a memory, a processor, and a control program of the virtual reality device stored in the memory and executable on the processor, and the control program of the virtual reality device, when executed by the processor, implements the steps of the control method of the virtual reality device as described above.
[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a control program of a virtual reality device is stored. When the control program of the virtual reality device is executed by a processor, the steps of the control method of the virtual reality device as described above are implemented.
[0038] The embodiments of the present invention provide a control method for a virtual reality device, a virtual reality device, and a storage medium. By embedding an FPGA module between a Camera module and a SOC, the FPGA module completes the line-by-line splicing of a single-frame image, and sends the spliced picture frame to the SOC, which can effectively release the computing power of the SOC. Since the power consumption of the FPGA is lower than that of the SOC, the image processing power consumption of the entire virtual reality device can be reduced, and the virtual reality device can be controlled to realize a full-color perspective function at a lower computing power cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of the architecture of the hardware operating environment of the virtual reality device involved in the embodiment of the present invention;
[0040] Figure 2 A schematic diagram of the internal connection architecture of a virtual reality device involved in an embodiment of the present invention;
[0041] Figure 3 It is a flowchart of a first embodiment of a method for controlling a virtual reality device of the present invention;
[0042] Figure 4 FIG. 4 is a flow chart of a second embodiment of a method for controlling a virtual reality device according to the present invention.
[0043] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0044] This application embeds an FPGA module between the Camera module and the SOC, and the FPGA module completes the line-by-line stitching of a single frame image, and sends the stitched frame to the SOC, which can effectively release the computing power of the SOC. Since the power consumption of the FPGA is lower than that of the SOC, the image processing power consumption of the entire virtual reality device can be reduced, and the virtual reality device can be controlled to achieve a full-color perspective function at a lower computing power cost.
[0045] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0046] As an implementation plan, Figure 1 The figure is a schematic diagram of the architecture of the hardware operating environment of the virtual reality device involved in the embodiment of the present invention.
[0047] like Figure 1 As shown, the virtual reality device may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.
[0048] Those skilled in the art will understand that Figure 1 The virtual reality device architecture shown in the figure does not constitute a limitation on the virtual reality device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0049] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a control program of a virtual reality device. The operating system is a program for managing and controlling the hardware and software resources of the virtual reality device, the control program of the virtual reality device, and the operation of other software or programs.
[0050] exist Figure 1 In the virtual reality device shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the control program of the virtual reality device stored in the memory 1005.
[0051] In this embodiment, the virtual reality device includes: a memory 1005, a processor 1001, and a control program of the virtual reality device stored in the memory and executable on the processor, wherein:
[0052] When the processor 1001 calls the control program of the virtual reality device stored in the memory 1005, the following operations are performed:
[0053] Determine the display content corresponding to the current pixel row according to the image frame data received by each of the input pins;
[0054] Determine whether the current pixel row is the last row corresponding to the current picture frame;
[0055] If yes, send the current picture frame to the SOC;
[0056] Otherwise, the next pixel row is used as the current pixel row, and the process jumps to executing the step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins.
[0057] When the processor 1001 calls the control program of the virtual reality device stored in the memory 1005, the following operations are performed:
[0058] Determine whether the current pixel row is the last row according to the display position corresponding to the display content in the current picture frame; or
[0059] Determine whether the current pixel row is the last row according to the receiving order corresponding to the image frame data.
[0060] When the processor 1001 calls the control program of the virtual reality device stored in the memory 1005, the following operations are performed:
[0061] Determine whether the display position is at the bottom of the current picture frame, and if so, determine that the current pixel row is the last row; or,
[0062] Determine whether the splicing sequence corresponding to the display position is the last splicing sequence corresponding to the current picture frame, and if so, determine that the current pixel row is the last row.
[0063] When the processor 1001 calls the control program of the virtual reality device stored in the memory 1005, the following operations are performed:
[0064] Determine the receiving order according to the receiving time when each of the input pins receives the image frame data, wherein the image frame data with an earlier receiving time has an earlier corresponding receiving order;
[0065] Determine whether the receiving order is the last receiving order corresponding to the image frame data, and if so, determine that the current pixel row is the last row.
[0066] When the processor 1001 calls the control program of the virtual reality device stored in the memory 1005, the following operations are performed:
[0067] Receive a long data packet sent by the Camera module to the input pin;
[0068] Unpacking the long data packet to obtain image frame data corresponding to the current pixel row, wherein the image frame data from different Camera modules have different field angles;
[0069] Splicing the image frame data in order from left to right to obtain spliced image data corresponding to the current pixel row, wherein the field of view corresponding to the spliced image data is greater than the field of view corresponding to each of the image frame data;
[0070] The display content corresponding to the current pixel row is determined according to the spliced image data.
[0071] When the processor 1001 calls the control program of the virtual reality device stored in the memory 1005, the following operations are performed:
[0072] Calling the parser in the MIPI-RX core;
[0073] The long data packet is unpacked by the parser to obtain the image frame data.
[0074] When the processor 1001 calls the control program of the virtual reality device stored in the memory 1005, the following operations are performed:
[0075] Calling the wrapper in the IP core;
[0076] Encapsulating the current picture frame through the encapsulator;
[0077] The packaged current picture frame is sent to the SOC via the MIPI-CSI output line.
[0078] Based on the hardware architecture of the virtual reality device based on the virtual reality device control technology, an embodiment of a control method of the virtual reality device of the present invention is proposed.
[0079] Reference Figure 2 , Figure 2 A schematic diagram of the internal connection architecture of a virtual reality device is shown. The virtual reality device includes multiple Camera modules, FPGA modules and SOCs. The Camera modules are connected to the input pins of the FPGA modules in a one-to-one correspondence, and the SOC is connected to the output end of the FPGA module.
[0080] Reference Figure 3 In a first embodiment, the control method of the virtual reality device comprises the following steps:
[0081] Step S10, determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins;
[0082] In this embodiment, a plurality of input pins are arranged on the FPGA module of the virtual reality device (hereinafter referred to as VR), and one input pin is connected to one Camera module, that is, the Camera module is connected to the input pin of the FPGA module in a one-to-one correspondence.
[0083] In this embodiment, multiple Camera modules of VR capture images simultaneously. Since the field of view of a single Camera module is limited, in order to match the display image captured by VR with the wearer's eye field of view, the images captured by multiple Camera modules need to be spliced together.
[0084] In this embodiment, the Camera module sends one line of data in a frame of image data (i.e., image frame data) to the FPGA module at a time. The FPGA module splices one line of image frame data at a time and determines the display content corresponding to a row of pixels of the spliced image frame data on the VR display screen.
[0085] It should be noted that the purpose of using single-row input splicing in this embodiment is that the FPGA module can filter out invalid frames in the image frame data input by the Camera module during the splicing process.
[0086] Step S20, determining whether the current pixel row is the last row corresponding to the current picture frame;
[0087] In this embodiment, after the display content of the current pixel row is determined, it is determined whether the current pixel row is the last row corresponding to the current picture frame.
[0088] It should be noted that the last row here refers to the last row of the current picture frame recognized by the VR device. There are many definitions of the order between pixel rows. How to define the "last row" will be explained in detail in subsequent embodiments and will not be repeated here.
[0089] Step S30: If yes, send the current picture frame to the SOC;
[0090] Step S40: otherwise, the next pixel row is used as the current pixel row, and the process jumps to the step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins.
[0091] In this embodiment, if the current pixel behavior is the last row recognized by the VR, it means that the display content of the current picture frame has been determined, and the FPGA module sends the spliced current picture frame to the SOC, and the SOC then projects the current picture frame onto the VR display screen to show it to the user.
[0092] In this embodiment, if the current pixel row is not the last row identified by VR, the next pixel row is used as the current pixel row, and the process jumps back to execute the action in step S10 until the display content of each pixel row of the current picture frame is determined.
[0093] In this embodiment, the next pixel row is characterized as a pixel row that is identified by VR and is after the current pixel row. It can be a pixel row that is adjacent to the current pixel row and located below the current pixel row on the display screen, or it can be a "next pixel row" defined based on other rules. This will also be described in detail in subsequent embodiments and will not be repeated here.
[0094] In addition, it should be noted that although the present embodiment processes one row of an image frame, since the internal circuits of the FPGA module are parallel, multiple rows of image frames can be processed simultaneously or almost simultaneously. Therefore, when the VR device is actually executing the solution, the entire image processing process is very fast.
[0095] In the technical solution provided in this embodiment, FPGA is used as the edge image coprocessor of SOC. On the one hand, the computing power of the main control chip SOC can be saved, and the data verification processing and image stitching of the Camera module input can be delegated to the coprocessor; on the other hand, FPGA adopts a low-power design, and data processing on FPGA will save more power than SOC. Finally, thanks to the parallelism of the internal circuits of the FPGA chip, it has the processing advantage of faster running speed compared to SOC in image data processing and algorithm execution.
[0096] Furthermore, in this embodiment, before step S10, the following steps are further included:
[0097] Step S50, receiving a long data packet sent by the Camera module to the input pin;
[0098] Step S60, unpacking the long data packet to obtain image frame data corresponding to the current pixel row, wherein the image frame data from different Camera modules have different viewing angles;
[0099] The step S10 comprises:
[0100] Step S11, stitching the image frame data in order from left to right to obtain stitched image data corresponding to the current pixel row, wherein the viewing angle corresponding to the stitched image data is greater than the viewing angle corresponding to each of the image frame data;
[0101] Step S12: determining the display content corresponding to the current pixel row according to the stitched image data.
[0102] Optionally, in this embodiment, the data sent by the Camera module through the input pin is long data packet data, and the FPGA module unpacks the long data packet data to obtain image frame data.
[0103] In this embodiment, the Camera module is horizontally arranged on the VR device, and the horizontal field of view angles of the collected image frame data are different.
[0104] For example, assume that the horizontal field of view of the picture captured by the VR device is 270 degrees, the leftmost picture is 0 degrees, and the rightmost picture is 270 degrees. The VR device is provided with three modules, namely, Camera module 1, Camera module 2, and Camera module 3. The horizontal field of view angle of the image frame data captured by Camera module 1 located on the leftmost side of the VR is [0, 90], the horizontal field of view angle of the image frame data captured by Camera module 2 is [90, 180], and the horizontal field of view angle of the image frame data captured by Camera module 3 is [180, 270]. When the picture frame data is displayed on the VR, the image frame data captured by the three Camera modules are spliced together to form spliced image frame data with a horizontal field of view covering the interval of [0, 270], which is used as the display content displayed on the VR display screen.
[0105] Of course, those skilled in the art will understand that when VR actually performs a stitching operation, there will be overlapping fields of view between the images captured by multiple Camera modules, and the above example is only used as an example to illustrate image stitching in this embodiment.
[0106] In this embodiment, after the splicing is completed to obtain the spliced image data, the display content corresponding to the current pixel row is determined according to the spliced image data.
[0107] In the technical solution provided in this embodiment, the FPGA module is used to complete the left-to-right stitching of the images captured by the Camera module, effectively releasing the computing power of the SOC. Since the power consumption of the FPGA is lower than that of the SOC, the image processing power consumption of the entire virtual reality device can be reduced, and the virtual reality device can be controlled to achieve image stitching at a lower computing power cost.
[0108] Furthermore, in this embodiment, step S60 includes:
[0109] Step S61, calling the parser in the MIPI-RX core;
[0110] Step S62, unpacking the long data packet through the parser to obtain the image frame data.
[0111] Optionally, in this embodiment, a MIPI-RX core is provided inside the FPGA module, and a parser is provided on the MIPI-RX core. When the unpacking step is performed, the long data packet is unpacked by the parser.
[0112] In the technical solution provided in this embodiment, by calling the parser in the MIPI-RX core of the FPGA chip to unpack the long data packet, the image frame data is obtained, so that the VR device can read and process the image data from the MIPI interface, realize high-speed data transmission and reliable data reception, thereby improving the processing speed of image data, reducing the response time, and further realizing the control of the virtual reality device to achieve screen splicing at a lower computing power cost.
[0113] Furthermore, in this embodiment, before step S30, the following steps are further included:
[0114] Step S70, calling the encapsulator in the IP core;
[0115] Step S80, encapsulating the current picture frame through the encapsulator;
[0116] The step S30 comprises:
[0117] Step S31, sending the packaged current picture frame to the SOC via the MIPI-CSI output line.
[0118] Optionally, in this embodiment, an IP core is provided inside the FPGA module, and an encapsulator is provided on the MIPI-RX core. Before the current picture frame is sent to the SOC, the current picture frame is encapsulated by the encapsulator.
[0119] In this embodiment, the FPGA output end is connected to the SOC via the MIPI-CSI output line. After the encapsulation action is performed, the encapsulated current picture frame is sent to the SOC via the MIPI-CSI output line.
[0120] In the technical solution provided in this embodiment, the encapsulated picture frame is sent to the SOC through the MIPI-CSI output line set on the output end of the FPGA. MIPI-CSI is a high-speed serial interface with the characteristics of fast image transmission speed, high resolution support, and strong signal anti-interference ability. It can reduce interference during data transmission, thereby ensuring the efficiency and accuracy of data. At the same time, by encapsulating and processing the picture frame through FPGA, real-time processing and optimization of data can be achieved, the response speed of SOC can be improved, and the virtual reality device can be controlled to achieve picture splicing at a lower computing power cost.
[0121] Furthermore, in this embodiment, as an optional implementation scheme, data is transmitted between the Camera module and the FPGA module based on the MIPI-CSI protocol. The MIPI-CSI protocol requires that the image frame data transmission process starts with short packet data (Start of Packet). During this process, the FPGA runs in low power mode. After transmitting several data frames, the long packet data (Long of Packet) is used as the main transmission object. During this process, the FPGA enters the high speed mode, that is, the relevant steps of calling the parser in the MIPI-RX core to unpack the long data packet in this embodiment are executed. Among them, the long data packet includes the data start bit, the packet header, the image data frame and the data end bit.
[0122] In the technical solution provided in this embodiment, the Camera module and the FPGA module transmit data based on the MIPI-CSI protocol. The MIPI-CSI protocol has the advantages of fast image transmission speed, high support resolution, strong signal anti-interference ability, etc. It can reduce interference during data transmission, thereby ensuring the efficiency and accuracy of data. At the same time, by encapsulating and processing the picture frame through FPGA, real-time processing and optimization of data can be achieved, the response speed of SOC can be improved, and the virtual reality device can be controlled to achieve picture splicing at a lower computing power cost.
[0123] Reference Figure 4 In the second embodiment, based on the first embodiment, step S20 includes:
[0124] Step S21, determining whether the current pixel row is the last row according to the display position corresponding to the display content in the current picture frame;
[0125] Alternatively, in step S22, determining whether the current pixel row is the last row according to the receiving order corresponding to the image frame data.
[0126] As an optional embodiment, this embodiment provides two different definition methods for how to define the "last row".
[0127] First, judging whether the current pixel row is the last row according to the display position of the display content in the picture frame. The display position of the picture frame corresponds to the display position shown to the user on the VR display screen.
[0128] Furthermore, the display position is determined in two specific ways:
[0129] (1) Determine whether the display position is at the bottom of the current picture frame. If so, determine that the current pixel row is the last row; otherwise, it is not.
[0130] Specifically, using this method to judge, the execution logic of the FPGA module is to splice the image frame data sent by the Camera module row by row from the top of the picture frame, in order from top to bottom. The current pixel row at the bottom of the current picture frame is the last row.
[0131] As an implementation scheme, a coordinate system may be defined on the image frame, and whether the display position is at the bottom may be determined by determining whether the display position is located at the bottom of the image frame on the coordinate system.
[0132] (2) Determine whether the splicing sequence corresponding to the display position is the last splicing sequence corresponding to the current picture frame.
[0133] Specifically, a splicing order is set for the received image frame data, and the image frame data is spliced in order during the splicing process. The splicing order may be different from the display position, rather than being spliced line by line from top to bottom.
[0134] Optionally, the stitching order can be associated with the size of the image frame data. The same frame of image frame data received is divided into sub-image frames of different pixel rows, and the sub-image frames are sorted according to the data size. The larger the data, the higher the corresponding stitching order (i.e., the rows with larger data are spliced preferentially). The sub-image frame with the smallest data has the last stitching order at the corresponding display position.
[0135] Optionally, the splicing order may also be associated with the receiving time of the image frame data and sorted in chronological order, and the splicing order of the display position corresponding to the last received image frame data is the last splicing order.
[0136] For example, assume that the pixel rows are set to A, B, C, and D in order from top to bottom, with a total of 4 pixel rows, corresponding to the first row, second row, third row, and fourth row on the VR display screen. Assuming that the respective splicing orders are determined to be: B, C, A, D, when the FPGA performs the image frame splicing operation, it first splices the second row, then the third row, then the first row, and finally jumps to splicing the fourth row.
[0137] 2. Determine whether the current pixel row is the last row according to the receiving sequence corresponding to the image frame data. The receiving sequence corresponding to the last received image frame data is the last receiving sequence.
[0138] Specifically, when VR captures each frame of image, each Camera module in it will capture the image, but there will be a slight transmission time difference when capturing and transmitting to the FPGA. Therefore, if this method is adopted, the image frame data received first will be spliced first, no matter where it is displayed, and the pixel row of the last received image frame data will be the last row.
[0139] In the technical solution provided in this embodiment, two dimensions are given for VR to identify whether the current pixel row is the last row, namely, the order of display positions and the order of receiving image frame data. If splicing is performed according to the order of display positions, the continuous image frame data displayed to the user after splicing will be smoother, and the picture seen by the user is not prone to quality degradation, and the virtual reality device is controlled at a lower computing power cost to achieve picture splicing while ensuring the quality of the output picture. If splicing is performed according to the receiving order, the data processing process for VR will be smoother, and the virtual reality device can be further controlled at a lower computing power cost to achieve picture splicing.
[0140] In addition, it can be understood by a person skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the virtual reality device to implement the process steps of the embodiment of the above method.
[0141] Therefore, the present invention also provides a computer-readable storage medium, which stores a control program of a virtual reality device. When the control program of the virtual reality device is executed by a processor, the control program of the virtual reality device implements the various steps of the control method of the virtual reality device described in the above embodiment.
[0142] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which may store program codes.
[0143] It should be noted that since the storage medium provided in the embodiment of the present application is the storage medium used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the method of the embodiment of the present application belong to the scope of protection of this application.
[0144] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0148] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.
[0149] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0150] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for controlling a virtual reality device, characterized in that: Applied to a virtual reality device, the virtual reality device includes at least two camera modules, an FPGA module and a SOC, the camera module is connected to the input pins of the FPGA module in a one-to-one correspondence, the SOC is connected to the output end of the FPGA module, and the control method of the virtual reality device includes the following steps: Determine the display content corresponding to the current pixel row according to the image frame data received by each of the input pins; Determine whether the current pixel row is the last row corresponding to the current picture frame; If yes, send the current picture frame to the SOC; Otherwise, the next pixel row is used as the current pixel row, and the process jumps to executing the step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins.
2. The control method of the virtual reality device according to claim 1, characterized in that: The step of determining whether the current pixel row is the last row corresponding to the current picture frame comprises: Determine whether the current pixel row is the last row according to the display position corresponding to the display content in the current picture frame; or Determine whether the current pixel row is the last row according to the receiving order corresponding to the image frame data.
3. The control method of the virtual reality device according to claim 2, characterized in that: The step of determining whether the current pixel row is the last row according to the display position corresponding to the display content corresponding to the current pixel row in the current picture frame comprises: Determine whether the display position is at the bottom of the current picture frame, and if so, determine that the current pixel row is the last row; or, Determine whether the splicing sequence corresponding to the display position is the last splicing sequence corresponding to the current picture frame, and if so, determine that the current pixel row is the last row.
4. The control method of the virtual reality device according to claim 2, characterized in that: Before the step of determining whether the current pixel row is the last row according to the receiving order corresponding to the image frame data, the step further includes: Determine the receiving order according to the receiving time when each of the input pins receives the image frame data, wherein the image frame data with an earlier receiving time has an earlier corresponding receiving order; The step of determining whether the current pixel row is the last row according to the receiving order corresponding to the image frame data comprises: Determine whether the receiving order is the last receiving order corresponding to the image frame data, and if so, determine that the current pixel row is the last row.
5. The control method of the virtual reality device according to claim 1, characterized in that: Before the step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins, the method further includes: Receive a long data packet sent by the Camera module to the input pin; Unpacking the long data packet to obtain image frame data corresponding to the current pixel row, wherein the image frame data from different Camera modules have different field angles; The step of determining the display content corresponding to the current pixel row according to the image frame data received by each of the input pins comprises: Splicing the image frame data in order from left to right to obtain spliced image data corresponding to the current pixel row, wherein the field of view corresponding to the spliced image data is greater than the field of view corresponding to each of the image frame data; The display content corresponding to the current pixel row is determined according to the spliced image data.
6. The control method of the virtual reality device according to claim 5, characterized in that: The FPGA module also includes a MIPI-RX core, and the step of unpacking the long data packet to obtain the image frame data corresponding to the current pixel row includes: Calling the parser in the MIPI-RX core; The long data packet is unpacked by the parser to obtain the image frame data.
7. The control method of the virtual reality device according to claim 1, characterized in that: The FPGA includes an IP core, the output end includes a MIPI-CSI output line, and before the step of sending the current picture frame to the SOC, the step further includes: Calling the wrapper in the IP core; Encapsulating the current picture frame through the encapsulator; The step of sending the current picture frame to the SOC comprises: The packaged current picture frame is sent to the SOC via the MIPI-CSI output line.
8. The control method of a virtual reality device according to any one of claims 1 to 7, characterized in that: Data is transmitted between the Camera module and the FPGA module based on the MIPI-CSI protocol.
9. A virtual reality device, characterized in that: The virtual reality device includes: a memory, a processor, and a control program of the virtual reality device stored in the memory and executable on the processor. When the control program of the virtual reality device is executed by the processor, the steps of the control method of the virtual reality device as described in any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a control program for a virtual reality device, and when the control program for the virtual reality device is executed by a processor, the steps of the method for controlling a virtual reality device according to any one of claims 1 to 8 are implemented.