Image processing method and apparatus for modular head-mounted device, and device and medium

Abstract

The present invention relates to the technical field of head-mounted devices. Disclosed are an image processing method and apparatus for a modular head-mounted device, and a device and a medium. The method is applied to a head-mounted display of a modular head-mounted device, and comprises: acquiring a notification time, wherein the notification time is located between the start time and end time of a current image data frame; when the notification time arrives, reading part of the current image data frame from a cache; and notifying a communication unit of sending part of the current image data frame to a calculation unit. The image processing method for a modular head-mounted device provided in the embodiments of the present invention can alleviate the problem of display lagging of the head-mounted display of the modular head-mounted device.

Description

Image processing method, device, equipment and medium for split head-mounted device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 31, 2023, with application number 202311431499.7 and invention name “Image processing method, device, equipment and medium for split-type head-mounted device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of head-mounted technology, and more particularly to an image processing method for a split head-mounted device, an image processing device for a split head-mounted device, a head-mounted device, a split head-mounted device, and a computer-readable storage medium. Background Art

[0003] Smart head-mounted devices are generally divided into two forms: integrated and split. Compared with integrated smart head-mounted devices, split smart head-mounted devices have better power consumption, battery life, weight, and heat dissipation. Therefore, split smart head-mounted devices have become a trend in market and technological development.

[0004] At present, the see-through function is generally added to smart head-mounted devices. Among them, the see-through function specifically allows users to see the surrounding environment and the virtual image projected in the smart head-mounted device. Common see-through functions are mainly achieved through two methods: optical see-through (OST) and video see-through (VST). Among them, video see-through has the advantages of a larger field of view, easier matching of the delay between the virtual world and the real world, and easier control of virtual and real fusion compared to optical see-through.

[0005] When implementing the video see-through function in a split-type smart head-mounted device, the camera on the headset captures an image of the real environment and transmits it via wired or wireless communication to the computing unit of the split-type smart head-mounted device. The computing unit then performs distortion and chromatic aberration correction on the received real environment image, as well as rendering processing such as virtual-reality fusion, to generate image data that includes both virtual and real elements. This image data is then transmitted via wired or wireless communication to the headset for display.

[0006] Based on the above, it can be seen that when a split-type smart head-mounted device implements video see-through, the real-world image must be transmitted to the computing unit, which then performs rendering processing before transmitting it to the headset for display. Furthermore, if the user's head or surrounding environment changes, the image displayed by the headset will experience a certain delay. This significantly reduces the user experience. Therefore, how to reduce the display delay of the headset has become a technical problem that needs to be solved urgently.

[0007] Summary of the Invention

[0008] An object of the present invention is to provide a new technical solution for image processing of a split-type head-mounted device.

[0009] According to a first aspect of the present invention, there is provided an image processing method for a split-type head-mounted device, the method being applied to a head-mounted display of the split-type head-mounted device, comprising:

[0010] Acquire a notification time, where the notification time is between a start time and an end time of the current frame image data;

[0011] When the notification time arrives, reading part of the frame image data in the current frame image data from the cache;

[0012] And, the communication unit is notified to send the partial frame image data to the calculation unit.

[0013] Optionally, the obtaining notification time includes:

[0014] Get the performance parameters of the application currently running on the split head-mounted device;

[0015] The notification time is determined according to the parameter information.

[0016] Optionally, the obtaining notification time includes:

[0017] Determine the storage duration of the current frame image data;

[0018] The notification time is determined according to the storage duration.

[0019] Optionally, the method further includes:

[0020] Provide an input entry for inputting notification time indication information;

[0021] In response to input from the input entry, determining notification time indication information;

[0022] The determining the notification time according to the storage duration includes:

[0023] The notification time is determined according to the storage duration and the notification time indication information.

[0024] Optionally, the performance parameter includes at least one of rendering performance, algorithm performance, and display performance.

[0025] Optionally, the notifying the communication unit to send the portion of the frame image data to the computing unit includes:

[0026] An interrupt signal is sent to the communication unit, where the interrupt signal is used to instruct the communication unit to send the portion of frame image data to the calculation unit.

[0027] Optionally, the sending an interrupt signal to the communication unit includes:

[0028] The camera is instructed to send a camera hardware interrupt signal to the communication unit.

[0029] According to a second aspect of the present invention, there is provided an image processing device for a split-type head-mounted device, which is applied to a head-mounted display of the split-type head-mounted device, comprising:

[0030] An acquisition module, configured to acquire a notification time, wherein the notification time is between a start time and an end time of the current frame image data;

[0031] a reading module, configured to read a portion of the current frame image data from the cache when the notification time arrives;

[0032] The notification module is used to notify the communication unit to send the portion of frame image data to the computing unit.

[0033] According to a third aspect of the present invention, a head-mounted display (HMD) of a split-type head-mounted device is provided, the HMD comprising the image processing device of the split-type head-mounted device as described in the second aspect; or

[0034] The head-mounted display includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the image processing method of the split head-mounted device as described in any one of the first aspects.

[0035] According to a fourth aspect of the present invention, a split-type head-mounted device is provided, the split-type head-mounted device comprising the head-mounted display according to the third aspect and a computing unit, wherein:

[0036] The computing unit is configured to receive partial frame image data sent by the head display, perform rendering processing on the partial frame image data to obtain rendered partial frame image data, and send the rendered partial frame image data to the head display so that the head display displays the rendered partial frame image data.

[0037] Optionally, the computing unit is connected to the head display via USB communication, and the computing unit is specifically configured to:

[0038] transmitting the partial frame image data sent by the head display to a rendering unit in the computing unit through a data transmission path, and having the rendering unit perform rendering processing on the partial frame image data to obtain rendered partial frame image data;

[0039] and, the rendering unit sending the rendered partial frame image data to the head-mounted display via a reverse path of the data transmission path;

[0040] The data transmission path is a path that passes through the USB Driver, Camera HAL, Camera Service, and Camera API in the computing unit in sequence.

[0041] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method of the split-type head-mounted device according to any one of the first aspects is implemented.

[0042] An embodiment of the present invention provides an image processing method for a split head-mounted device. This method, applied to a head-mounted display (HMD) of a split head-mounted device, includes: obtaining a notification time, the notification time being between the start and end times of the current frame of image data; upon arrival of the notification time, reading a portion of the current frame of image data from a cache; and notifying a communication unit to send the portion of the frame of image data to a computing unit. This method enables the cache to store image data, the communication unit to transmit image data, the computing unit to render, and the HMD to refresh the screen. In other words, storage, transmission, rendering, and screen refreshing are performed simultaneously. This improves the HMD display latency issue caused by conventional techniques, which require waiting for the camera to capture and transmit a complete frame of image data to the cache, i.e., a complete frame of image data is stored in the cache before the processor notifies the communication unit to send the complete frame of image data to the computing unit, which then renders the complete frame of image data and sends the feedback to the HMD. Based on this, the image processing method for a split head-mounted device provided by the embodiment of the present invention can alleviate the display latency issue of the HMD of a split head-mounted device.

[0043] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0045] FIG1 is a block diagram 1 of a hardware configuration of a head-mounted display according to an image processing method for a split head-mounted device provided by an embodiment of the present invention;

[0046] FIG2 is a schematic flow chart of an image processing method for a split-type head-mounted device according to an embodiment of the present invention;

[0047] FIG3 is a data flow diagram of a split-type head-mounted device provided according to an embodiment of the present invention;

[0048] FIG4 is a schematic structural diagram of an image processing device for a split-type head-mounted device provided according to an embodiment of the present invention;

[0049] 5 is a block diagram 2 of the hardware configuration of an integrated head-mounted device according to an image processing method for a split head-mounted device provided by an embodiment of the present invention;

[0050] FIG6 is a data flow diagram of a computing unit provided according to an embodiment of the present invention.

[0051] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0054] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme in which A and B are satisfied at the same time. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0055] FIG1 is a block diagram 1 of a hardware configuration of a head-mounted display in a split-type head-mounted device according to an image processing method of the split-type head-mounted device provided by an embodiment of the present invention.

[0056] Among them, the head display 1000 is exemplarily an AR, MR, VR or XR head-mounted display (such as glasses, helmets, etc.) device.

[0057] The head-mounted display (HMD) 1000 may include a processor 1100, a memory 1200, an interface unit 1300, a communication unit 1400, a display unit 1500, an input unit 1600, a speaker 1700, a microphone 1800, and a camera 1900, among others. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), or the like. The memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), or a non-volatile memory such as a hard disk. The interface unit 1300 may include, for example, a USB port or a headphone jack. The communication unit 1400 may be capable of wired or wireless communication. The display unit 1500 may be, for example, an LCD display or a touchscreen display. The input unit 1600 may include, for example, a touchscreen or buttons. Users may input and output voice information via the speaker 1700 and microphone 1800.

[0058] Although multiple units are shown for the head display 1000 in Figure 1, the present invention may only involve some of the units. For example, the head display 1000 only involves the memory 1200, the processor 1100, the display unit 1500, and the camera 1900.

[0059] In an embodiment of the present invention, the memory 1200 of the head-mounted display 1000 is used to store instructions, which are used to control the processor 1100 to execute the image processing method for the split head-mounted device provided in an embodiment of the present invention.

[0060] In the above description, a person skilled in the art can design instructions according to the solution disclosed in the present invention. How instructions control the operation of a processor is well known in the art and will not be described in detail here.

[0061] An embodiment of the present invention provides an image processing method for a split-type head-mounted device, which is applied to a head-mounted display of the split-type head-mounted device.

[0062] It should be noted that the split head-mounted device includes a head display and a computing unit. The computing unit has an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present invention.

[0063] As shown in FIG2 , an image processing method for a split-type head-mounted device provided by an embodiment of the present invention includes the following steps S2100 to S2300 .

[0064] S2100. Obtain notification time.

[0065] The notification time is between the start time and the end time of the current frame image data.

[0066] In an embodiment of the present invention, the notification time is the time for reading part of the frame image data (less than one frame) belonging to the current frame image data from the cache, and notifying the communication unit to send the read part of the frame image data of the current frame image data to the computing unit.

[0067] In one embodiment of the present invention, the notification time can be described by a frame length. For example, if the notification time is the time corresponding to 1 / 2 frame, this means that the notification time is the moment when the camera transmits the image data of 1 / 2 frame to the buffer, or the moment when the image data of 1 / 2 frame is stored in the buffer.

[0068] The notification time is between the start time and the end time of the current frame image data. It is understandable that the notification time does not include the start time and the receiving time of the current frame image data.

[0069] Furthermore, the start time of the current frame image data is usually referred to as the start of frame (SOF) time, and the end time of the current frame image data is usually referred to as the end of frame (EOF) time.

[0070] In the embodiment of the present invention, the notification time may be obtained in at least three ways as follows.

[0071] In the first method, the above S2100 can be specifically implemented through the following S2110 and S2111.

[0072] S2110. Obtain parameter information of performance parameters of an application currently running on the split head-mounted device.

[0073] In one embodiment of the present invention, the performance parameters may include at least one of rendering performance, algorithm performance, and display performance. The algorithm performance parameters include computing power, power consumption, etc. The display performance parameters include resolution, etc.

[0074] Of course, the performance parameters may also include other parameters, which are not limited in the present invention.

[0075] It should be noted that, in the embodiment of the present invention, there is no limitation on how to determine the currently running application and how to obtain parameter information of the performance parameters of the currently running application.

[0076] S2111. Determine the notification time based on the parameter information.

[0077] In an embodiment of the present invention, when it is determined based on parameter information that the currently running application needs to quickly display the current frame image data, the interval between notification times is determined to be small. In this way, the rapid display of the currently running application can be guaranteed by increasing the reading and transmission frequency of the current frame image data.

[0078] In one embodiment of the present invention, a correspondence between application performance parameter information and notification times can be established in advance based on experience, and this correspondence can be stored in the split head-mounted device. Based on this, the notification time corresponding to the performance parameter information of the currently running application can be found from the aforementioned correspondence, and the notification time corresponding to the performance parameter information that is closest to the performance parameter information can be used as the notification time in S2111 above.

[0079] Through the above S2110 and S2111, the adaptive notification time can be automatically obtained according to the parameter information of the performance parameters of the currently running application, thereby eliminating the need to manually specify the notification time, which improves the intelligence of the image processing method of the split head-mounted device provided in an embodiment of the present invention.

[0080] In the second method, the above S2100 can be specifically implemented through the following S2120 and S2121.

[0081] S2120: Determine the storage duration of the current frame image data.

[0082] In the embodiment of the present invention, the storage duration of the current frame image data specifically refers to the duration required for storing the complete frame image data of the current frame in the buffer.

[0083] It is understood that the storage duration of the current frame image data is the same as the storage duration of the historical frame image data. Therefore, the storage duration of the historical frame image data (the storage duration of the historical frame image data is known) can be determined as the storage duration of the current frame image. The storage duration of the historical frame image data is specifically the duration corresponding to the start time and end time of the historical frame image data.

[0084] Alternatively, if the currently running application is determined, the storage duration of the image frame data of the currently running application is also fixed and known. Based on this, the storage duration of the image frame data of the currently running application can be directly read as the storage duration of the current frame image data.

[0085] S2121. Determine the notification time based on the storage duration.

[0086] In this embodiment of the present invention, taking the storage duration N as an example, the notification time can be determined as N / M frame lengths, where M is greater than 1 and can be pre-set and stored based on experience. It should be noted that N / M frame lengths specifically refer to the time it takes to store N / M frame lengths of image data in the cache, or the time it takes for the camera to capture N / M frame lengths of image data.

[0087] Through the above S2120 and S2121, the notification time can be automatically determined without manual designation of the notification time, which improves the intelligence of the image processing method of the split head-mounted device provided in the embodiment of the present invention.

[0088] In a third embodiment, the image processing method for a split head-mounted device provided by an embodiment of the present invention further includes the following S2130 and S2131.

[0089] S2130: Provide an input entry for inputting notification time indication information.

[0090] In an embodiment of the present invention, the input port is used by the user to input notification time indication information. The notification time indication information is used to indicate the notification time to the split-type head-mounted device.

[0091] In an example, the notification time indication information may be illustratively 1 / M, where M is greater than 1.

[0092] S2131. In response to input from an input entry, determine notification time indication information.

[0093] In the embodiment of the present invention, the user inputs the notification time indication information in the input entrance, and the split-type head-mounted device receives the notification indication information through the input entrance.

[0094] Based on the above S2130 and S2131, the above S2100 is implemented through the following S2132.

[0095] S2132: Determine the notification time according to the storage duration and the notification time indication information.

[0096] In the embodiment of the present invention, taking the notification time indication information as 1 / M as an example, the specific implementation of the above S2132 may be to determine the notification time as N / M frame length.

[0097] In the third method described above, the user may indicate the notification time. In this way, the image processing method for the integrated head-mounted device provided by the embodiment of the present invention can meet the personalized needs of the user.

[0098] S2200 : When the notification time arrives, read part of the current frame image data from the cache.

[0099] S2300 , instruct the communication unit to send part of the frame image data to the computing unit.

[0100] The communication unit may be a WIFI unit or a USB unit.

[0101] In traditional technology, a camera captures an image of, for example, the environment, and sends the captured image data in real time to a cache for storage. When the camera captures and transmits a complete frame of image data to the cache, i.e., when a complete frame of image data is stored in the cache, the processor reads the complete frame of image data from the cache. When the camera does not capture and transmit a complete frame of image data to the cache, i.e., when a complete frame of image data is not stored in the cache, the processor waits for the current frame of image data. After reading the complete frame of image data from the cache, the processor notifies the communication unit to send the complete frame of image data to the computing unit. The computing unit renders the complete frame of image data sent by the processor of the headset and sends the rendered complete frame of image data to the headset, which is then displayed on the screen.

[0102] In an embodiment of the present invention, a camera captures images of, for example, the environment, and sends the captured image data to a cache in real time for storage. When the notification time arrives, the processor reads partial frame image data from the cache. Because the notification time is between the start time and the end time of the current frame image data, before a complete frame of image data is stored in the cache, the processor begins reading partial frame image data from the cache and notifies the communication unit to send the partial frame image data to the computing unit. Based on the notification, the communication unit sends the partial frame image data to the computing unit. The computing unit renders the partial frame image data sent by the processor of the head display, and sends the rendered partial frame image data to the head display for display on the head display screen. Compared to traditional technologies, this can achieve a more immediate display.

[0103] It should be noted that when the camera captures and transmits a complete frame of image data, that is, when a complete frame of image data is stored in the cache, the processor will read the current frame image data from the cache, as in conventional technology, and notify the communication unit to transmit it to the computing unit. Since the processor has already read part of the frame image data, the processor will read the remaining frame image data that has not been read in the current frame image data, and notify the communication unit to transmit it to the computing unit. Furthermore, the communication unit sends the remaining frame image data to the computing unit. The computing unit renders the remaining frame image data sent by the communication unit of the head display, and sends the rendered remaining frame image data to the head display for display on the head display screen.

[0104] Taking a notification time of 1 / 2 frame length as an example, when the camera captures and transmits image data for 1 / 2 of the current frame to the cache, i.e., when the cache stores image data for 1 / 2 of the current frame, the processor reads image data belonging to the current frame from the cache, i.e., the 1 / 2 frame of image data, as partial frame image data for the current frame. The processor instructs the communication unit to transmit the read 1 / 2 frame of image data to the computing unit. The communication unit sends the read 1 / 2 frame of image data to the computing unit. The computing unit renders the 1 / 2 frame of image data and sends the rendered 1 / 2 frame of image data to the head-mounted display for display. Furthermore, when the camera captures and transmits the image data for the current frame to the cache, i.e., when the cache stores the remaining 1 / 2 frame of image data, the processor continues to read the remaining 1 / 2 frame of image data from the cache and instructs the communication unit to send it to the computing unit. The communication unit sends the remaining 1 / 2 frame of image data to the computing unit. The computing unit renders the remaining half frame of image data and sends the rendered half frame of image data to the head-mounted display, which then refreshes the screen to display it. In this way, the head-mounted display completes the display of the full frame of image.

[0105] Based on the above content, it can be seen that the image processing method for a split head-mounted device provided by an embodiment of the present invention can achieve: the cache stores image data while the communication unit transmits image data while the computing unit renders while the head-mounted display refreshes the screen. That is, storage, transmission, rendering, and screen refresh are performed simultaneously. This improves the head-mounted display delay problem caused by the need to wait for the camera to capture and transmit a complete frame of image data to the cache, that is, a complete frame of image data is stored in the cache, and the processor notifies the communication unit to send the complete frame of image data to the computing unit, and the computing unit renders the complete frame of image data and feeds back to the head-mounted display. Based on this, the image processing method for a split head-mounted device provided by an embodiment of the present invention can alleviate the display delay problem of the head-mounted display of a split head-mounted device.

[0106] An embodiment of the present invention provides an image processing method for a split head-mounted device. This method, applied to a head-mounted display (HMD) of a split head-mounted device, includes: obtaining a notification time, the notification time being between the start and end times of the current frame of image data; upon arrival of the notification time, reading a portion of the current frame of image data from a cache; and notifying a communication unit to send the portion of the frame of image data to a computing unit. This method enables the cache to store image data, the communication unit to transmit image data, the computing unit to render, and the HMD to refresh the screen. In other words, storage, transmission, rendering, and screen refreshing are performed simultaneously. This improves the HMD display latency issue caused by conventional techniques, which require waiting for the camera to capture and transmit a complete frame of image data to the cache, i.e., a complete frame of image data is stored in the cache before the processor notifies the communication unit to send the complete frame of image data to the computing unit, which then renders the complete frame of image data and sends the feedback to the HMD. Based on this, the image processing method for a split head-mounted device provided by the embodiment of the present invention can alleviate the display latency issue of the HMD of a split head-mounted device.

[0107] In one embodiment of the present invention, the above S2300 can be specifically implemented through the following S2310.

[0108] S2310: Send an interrupt signal to the communication unit.

[0109] The interrupt signal is used to instruct the communication unit to send part of the frame image data to the calculation unit.

[0110] In one embodiment of the present invention, the interrupt signal may be a hardware interrupt signal or a software interrupt signal.

[0111] In the embodiment of the present invention, an interrupt signal is sent to the communication unit to notify the communication unit to send part of the frame image data to the calculation unit. In this way, efficient and simple notification of the communication unit can be achieved.

[0112] In one embodiment of the present invention, the above-mentioned S2310 can be implemented through the following S2310-1.

[0113] S2310-1. Instruct the camera to send a camera hardware interrupt signal to the communication unit.

[0114] In this embodiment of the present invention, the camera is instructed to send a camera hardware interrupt signal to the communication unit, thereby notifying the communication unit to send a portion of the current frame image data to the computing unit. This allows for highly real-time and reliable notification of the image processing unit.

[0115] In conjunction with the above embodiment, taking the notification time as 1 / 2 frame length and the current frame image data as the Nth frame as an example, the data flow of the split head-mounted device provided by the embodiment of the present invention can be shown as Figure 3. Specifically, when the partial frame image data of the current frame of 1 / 2 frame is stored in the buffer (such as the Camera Buffer), the camera sends a camera hardware interrupt signal to the communication unit to notify the communication unit to transmit the 1 / 2 frame image data in the buffer to the computing unit.

[0116] An embodiment of the present invention further provides an image processing device for a split head-mounted device, which is applied to a head-mounted display of the split head-mounted device, as shown in FIG4 , and includes:

[0117] An acquisition module 410 is configured to acquire a notification time, where the notification time is between a start time and an end time of the current frame image data;

[0118] A reading module 420 is configured to read a portion of the current frame image data from a cache when the notification time arrives;

[0119] The notification module 430 is configured to notify the communication unit to send the portion of frame image data to the computing unit.

[0120] The image processing device for a split head-mounted device provided by an embodiment of the present invention can achieve the following: the cache stores image data while the communication unit transmits image data while the computing unit renders while the head-mounted display refreshes the screen. That is, storage, transmission, rendering, and screen refresh are performed simultaneously. This improves the head-mounted display delay problem caused by the need to wait for the camera to capture and transmit a complete frame of image data to the cache, that is, a complete frame of image data is stored in the cache before the processor notifies the communication unit to send the complete frame of image data to the computing unit, and the computing unit renders the complete frame of image data and feeds back to the head-mounted display. Based on this, the image processing method for a split head-mounted device provided by an embodiment of the present invention can alleviate the display delay problem of the head-mounted display of a split head-mounted device.

[0121] In one embodiment of the present invention, the acquisition module 410 is specifically configured to:

[0122] Get the performance parameters of the application currently running on the split head-mounted device;

[0123] The notification time is determined according to the parameter information.

[0124] In one embodiment of the present invention, the acquisition module 410 is specifically configured to:

[0125] Determine the storage duration of the current frame image data;

[0126] The notification time is determined according to the storage duration.

[0127] In one embodiment of the present invention, the image processing device 400 of the integrated head-mounted device further includes:

[0128] Providing a module for providing an input entry for inputting notification time indication information;

[0129] a response module, configured to determine notification time indication information in response to input from the input entry;

[0130] The acquisition module 410 is specifically used for:

[0131] The notification time is determined according to the storage duration and the notification time indication information.

[0132] In one embodiment of the present invention, the performance parameter includes at least one of rendering performance, algorithm performance, and display performance.

[0133] In one embodiment of the present invention, the communication module 430 is specifically configured to:

[0134] An interrupt signal is sent to the communication unit, where the interrupt signal is used to instruct the communication unit to send the portion of frame image data to the calculation unit.

[0135] In one embodiment of the present invention, the communication module 430 is specifically configured to:

[0136] The camera is instructed to send a camera hardware interrupt signal to the communication unit.

[0137] An embodiment of the present invention further provides a head-mounted display 500 of a split-type head-mounted device, the head-mounted display 500 including any one of the image processing devices 400 of the split-type head-mounted device provided in the above device embodiments;

[0138] Alternatively, as shown in Figure 5, the head display 500 includes a memory 510 and a processor 520, the memory 510 is used to store computer instructions, and the processor 520 is used to call the computer instructions from the memory 510 to execute the image processing method of the split head-mounted device as described in any one of the above method embodiments.

[0139] An embodiment of the present invention further provides a split head-mounted device, which includes any head-mounted display provided in the above-mentioned head-mounted display embodiments and a computing unit.

[0140] The computing unit is configured to receive partial frame image data sent by the head display, perform rendering processing on the partial frame image data to obtain rendered partial frame image data, and send the rendered partial frame image data to the head display so that the head display displays the rendered partial frame image data.

[0141] In one embodiment of the present invention, the computing unit is connected to the head display via USB communication, and the computing unit is specifically configured to:

[0142] transmitting the partial frame image data sent by the head display to a rendering unit in the computing unit through a data transmission path, and having the rendering unit perform rendering processing on the partial frame image data to obtain rendered partial frame image data;

[0143] and, the rendering unit sending the rendered partial frame image data to the head-mounted display via a reverse path of the data transmission path;

[0144] The data transmission path is a path that passes through the USB Driver, Camera HAL, Camera Service, and Camera API in the computing unit in sequence.

[0145] In the embodiment of the present invention, the rendering unit is specifically an application APP running in the computing unit that can perform image rendering.

[0146] The reverse path of the data transmission path is the path that passes through the Camera API, Camera Service, Camera HAL, and USB Driver in sequence.

[0147] The process of the computing unit receiving the partial frame image data sent by the head display, performing rendering processing on the partial frame image data to obtain rendered partial frame image data, and sending the rendered partial frame image data to the head display can be shown in FIG6 . Specifically:

[0148] The USB Driver receives 1 / 2 frame image data and sends it to the Camera HAL. The Camera HAL implements the conversion from the hardware interface to the software interface to pass the 1 / 2 frame image data received from the hardware USB Driver to the software level. The Camera Service is responsible for the interaction between the upper-layer application and the Camera HAL, helping the upper-layer application to implement operations at the hardware layer. The Camera Service continues to send the 1 / 2 frame image data to the app through the Camera API. The app renders the 1 / 2 frame image data. The app sends the rendered 1 / 2 frame image data to the headset through the path of Camera API→Camera Service→Camera HAL→USB Driver.

[0149] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image processing method for a split-type head-mounted device according to any one of the above method embodiments is implemented.

[0150] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0151] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0152] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0153] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0154] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0156] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0157] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent. It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0158] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0159] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, television, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0160] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. An image processing method for a split head-mounted device, characterized in that: The head display applied to the split head-mounted device comprises: Acquire the notification time, where the notification time is between the start time and the end time of the current frame image data; When the notification time arrives, reading part of the frame image data in the current frame image data from the cache; And, the communication unit is notified to send the part of the frame image data to the calculation unit.

2. The method according to claim 1, characterized in that: The acquisition notification time includes: Get the performance parameter information of the application currently running on the split head mounted device; The notification time is determined according to the parameter information.

3. The method according to claim 1, characterized in that The acquisition notification time includes: Determine the storage duration of the current frame image data; The notification time is determined according to the storage duration.

4. The method according to claim 3, characterized in that: The method further comprises: Providing an input entry for inputting notification time indication information; In response to input of the input entry, determining notification time indication information; The determining the notification time according to the storage duration includes: The notification time is determined according to the storage duration and the notification time indication information.

5. The method according to claim 2, characterized in that: The performance parameters include: at least one of rendering performance, algorithm performance, and display performance.

6. The method according to claim 1, characterized in that The notifying communication unit to send the part of the frame image data to the computing unit includes: Send an interrupt signal to the communication unit, the interrupt signal is used to instruct the communication unit to The partial frame image data is sent to the computing unit.

7. The method according to claim 6, characterized in that The sending an interrupt signal to the communication unit comprises: The camera is instructed to send a camera hardware interrupt signal to the communication unit.

8. An image processing device for a split head-mounted device, characterized in that: The head display applied to the split head-mounted device comprises: An acquisition module, used for acquiring a notification time, wherein the notification time is between a start time and an end time of the current frame image data; A notification module, configured to read part of the frame image data in the current frame image data from the cache when the notification time arrives; The notification module is used to notify the communication unit to send the part of the frame image data to the calculation unit.

9. A head display of a split head-mounted device, characterized in that: The head display comprises the image processing device of the split head-mounted device according to claim 8; or The head display includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the image processing method for the split head-mounted device according to any one of claims 1-7.

10. A split head-mounted device, characterized in that: The split head mounted device comprises the head mounted display and a computing unit as claimed in claim 9, wherein: The computing unit is used to receive partial frame image data sent by the head display, perform rendering processing on the partial frame image data to obtain rendered partial frame image data, and send the rendered partial frame image data to the head display so that the head display displays the rendered partial frame image data.

11. The split head-mounted device according to claim 10, characterized in that: The computing unit is connected to the head display via USB communication, and the computing unit is specifically used for: The partial frame image data sent by the head display is transmitted to the rendering unit in the computing unit through the data transmission path, and the rendering unit performs rendering processing on the partial frame image data. To the rendered partial frame image data; And, the rendering unit sends the rendered partial frame image data to the head display through a reverse path of the data transmission path; The data transmission path is a path that passes through the USB Driver, Camera HAL, Camera Service, and Camera API in the computing unit in sequence.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the image processing method of the split head-mounted device according to any one of claims 1-7 is implemented.

Images