XR equipment

Abstract

The invention discloses XR equipment, and belongs to the technical field of augmented reality. The XR equipment comprises a camera module and an SOC, and the SOC comprises an image signal processor and a central processing unit. The camera module is used for collecting a first image, extracting a second image from the first image, generating a metadata table corresponding to the first image and sending the metadata table and the second image to an image signal processor, and the metadata table is used for representing the validity of pixel units in the first image. And the image signal processor is used for sending the metadata table and the second image sent by the camera module to the central processor. And the central processing unit is used for executing corresponding operation based on the metadata table and the second image sent by the image signal processor.

Description

Technical Field

[0001] The present application belongs to the field of extended reality technology, and specifically relates to an XR device. Background Art

[0002] Extended Reality (XR) devices can combine reality and virtuality through computer technology, bringing users an "immersive feeling" of seamless transition between the virtual world and the real world. Current XR devices can enable users to interact with virtual interfaces through a combination of gesture recognition and eye tracking.

[0003] In the related technology, when the user interacts with the virtual interface displayed by the XR device through gestures and eyes, the XR device can collect the user's eye images and hand images through the camera module, and send the eye images and hand images collected by the camera module to the image signal processor (Image Signal Processor, ISP) through the mobile industry processor interface (Mobile Industry Processor Interface, MIPI). The ISP then sends the eye images and hand images to the central processing unit (Central Processing Unit, CPU). The CPU determines the user's interactive input to the virtual interface displayed by the XR device based on the eye images and hand images, and performs operations corresponding to the interactive input.

[0004] However, since the camera module captures original (RAW) images, the data volume of RAW images is usually large, while the transmission rate of MIPI is limited, the process of transmitting the images captured by the camera module to the ISP takes a long time, and the power consumption of transmitting the images is also relatively large. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an XR device that can reduce the time consumed in transmitting images, improve the efficiency of transmitting images, and reduce the power consumption of transmitting images.

[0006] In the first aspect, an embodiment of the present application provides an XR device, which includes a camera module and a system-on-a-Chip (SOC), and the SOC includes an image signal processor and a central processing unit. The above-mentioned camera module is used to capture a first image, extract a second image from the first image, and generate a metadata table corresponding to the first image, and send the metadata table and the second image to the image signal processor, and the metadata table is used to indicate the validity of the pixel unit in the first image. The above-mentioned image signal processor is used to send the metadata table and the second image sent by the camera module to the central processing unit. The above-mentioned central processing unit is used to perform corresponding operations based on the metadata table and the second image sent by the image signal processor.

[0007] In an embodiment of the present application, the XR device includes a camera module and a SOC, and the SOC includes an image signal processor and a central processing unit. The camera module is used to capture a first image, extract a second image from the first image, generate a metadata table corresponding to the first image, and send the metadata table and the second image to the image signal processor, and the metadata table is used to indicate the validity of the pixel unit in the first image. The image signal processor is used to send the metadata table and the second image sent by the camera module to the central processing unit. The central processing unit is used to perform corresponding operations based on the metadata table and the second image sent by the image signal processor. In this way, after the camera module captures the first image, it does not send the first image directly to the image signal processor, but instead sends the second image extracted from the first image and the generated metadata table to the image signal processor. Since the second image is extracted from the first image, the second image is smaller than the first image, and the metadata table uses one data to represent a pixel unit in the first image. Compared with using three-dimensional or four-dimensional data to represent a pixel unit in the first image, the amount of data is greatly reduced. It can be seen that the sum of the data amounts of the second image and the metadata table is smaller than that of the first image. Therefore, the amount of data transmitted from the camera module to the image signal processor is reduced, so the time consumption of the transmission process will also be reduced, the resources required for transmitting data will also be reduced, and the power consumption of transmitting data will also be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of the structure of an XR device in the related art;

[0009] Figure 2 is a schematic diagram of the structure of an XR device provided in an embodiment of the present application;

[0010] Figure 3 is a schematic diagram of an image provided in an embodiment of the present application;

[0011] Figure 4 is a schematic diagram of a metadata table provided in an embodiment of the present application;

[0012] Figure 5 is a schematic diagram of a process for processing an image provided by an embodiment of the present application;

[0013] Figure 6 It is a schematic diagram of the time consumption of data transmission provided in an embodiment of the present application;

[0014] Figure 7 It is a flowchart of the image processing method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0016] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0017] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0018] First, the technical terms involved in the embodiments of the present application are explained.

[0019] XR Headsets (Extended Reality Headsets): Wearable devices that combine Virtual Reality (VR) and Augmented Reality (AR) technologies.

[0020] Light Emitting Diode (LED): It is a commonly used light-emitting device that releases energy through the recombination of electrons and holes to emit light. It is widely used in the field of lighting.

[0021] SOC: is a technology that integrates all components of a computer or other electronic system into a single chip.

[0022] Frames Per Second (FPS): This is a definition in the field of graphics. Generally speaking, it refers to the number of frames in an animation or video. FPS measures the amount of information used to save and display dynamic video. The more frames per second, the smoother the displayed action will be.

[0023] RAW: The original meaning of RAW is "unprocessed". It can be understood that the RAW image is the original data that the image sensor converts the captured light source signal into a digital signal.

[0024] MIPI: The abbreviation of "Mobile Industry Processor Interface" in English, is the abbreviation of mobile industry processor interface, a set of open standards and specifications developed by the Mobile Industry Processor Interface Alliance.

[0025] meta: Metadata, also known as intermediary data or relay data, is data about data. It is mainly information describing data properties, and is used to support functions such as indicating storage location, historical data, resource search, and file records.

[0026] YUV: It is a color encoding method. YUV data consists of three components: Y, U, and V. YUV usually refers to YCbCr, where Y represents brightness (Luminance or Luma), Cb (U) represents the blue chromaticity component, and Cr (V) represents the red chromaticity component.

[0027] RGGB: It is a commonly used color filter array (CFA) arrangement, especially in image sensors, such as complementary metal oxide semiconductor (CMOS) sensors in digital cameras and mobile phone cameras. The RGGB arrangement is a form of Bayer color filter array (Bayer CFA), which captures color images by arranging four color filters of red (R), green (G), green (G), and blue (B) in a specific pattern above the sensor pixels.

[0028] Next, the structure of the XR device in the related art is described. Figure 1 As shown, Figure 1 1 is a schematic diagram of the structure of an XR device 10 provided in the related art. The XR device 10 may include a camera module 11 and a SOC 12, and the SOC 12 may include an ISP 121 and a CPU 122. In addition, the XR device 10 is also installed with a hand-eye interaction APP.

[0029] In one example, the camera module 11 is used to capture images from a camera sensor and send the captured images to ISP121. The ISP121 is used to receive images sent by the camera module 11, perform format conversion on the received images, and then send the format-converted images to CPU122. The CPU122 is used to receive the format-converted images sent by ISP121, and a hand-eye algorithm is built into the CPU122. Through the hand-eye algorithm, the user's input to the hand-eye interaction APP in the XR device 10 can be determined based on the received format-converted images. The hand-eye algorithm may include a cutout sub-algorithm, which is used to cut out images from the format-converted images. The CPU122 is also used to perform corresponding operations based on the user's input to the hand-eye interaction APP in the XR device 10 determined by the hand-eye algorithm, and display the interface of the hand-eye interaction APP after the operation is performed.

[0030] The XR device provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0031] Figure 2 Schematic diagram of the structure of the XR device provided in the embodiment of the present application. Figure 2 As shown, the XR device 20 provided in the embodiment of the present application may include a camera module 21 and a SOC 22 , and the SOC 22 may include an image signal processor 221 and a central processing unit 222 .

[0032] In an embodiment of the present application, the camera module 21 is used to capture a first image, extract a second image from the first image, generate a metadata table corresponding to the first image, and send the metadata table and the second image to the image signal processor 221.

[0033] In the embodiment of the present application, the XR device 20 may be a device with XR functions. For example, the XR device 20 may be an AR device, an MR virtual reality device, an XR head display device, a cycling helmet display device, a head-up display (HUD) device, etc.

[0034] In the embodiment of the present application, the camera module 21 may be referred to as a camera module. The camera module 21 may include an infrared camera, a color image camera, a depth camera, and an eye tracking camera.

[0035] Optionally, in the embodiment of the present application, the infrared camera is mainly used for environmental perception and motion capture. By capturing infrared signals of hand movements through the infrared camera, the XR device 20 can track the user's hand position and movement trajectory in real time, thereby achieving more natural and intuitive gesture interaction.

[0036] Optionally, in the embodiment of the present application, the color image camera is mainly used to collect external effects and provide real-time stereoscopic vision for the display module inside the XR device 20. At the same time, the color image camera can also be used to capture facial expressions, so that the XR device 20 can capture the user's facial expressions and reconstruct them in a virtual environment.

[0037] Optionally, in an embodiment of the present application, the above-mentioned depth camera mainly captures the depth information of the user's hand area to provide the user with a more accurate gesture tracking and interactive experience.

[0038] Optionally, in an embodiment of the present application, the above-mentioned eye tracking camera mainly utilizes technologies such as the pupil corneal reflection vector method to capture high-resolution images of the eyes and locate the positions of the pupil and corneal reflection points in real time, thereby estimating the user's line of sight direction and landing point, so that the XR device 20 can better understand the user's visual focus and intentions, thereby providing a more personalized interactive experience.

[0039] It should be noted that the number of the above-mentioned eye tracking cameras can be two, and one eye tracking camera is used to capture the image of one eye.

[0040] Optionally, in the embodiment of the present application, when the user uses the XR device 20, the eye tracking camera of the camera module 21 can collect the user's eye image in real time to obtain a first image. In addition, the eye image can be two images, one eye image includes one eye, that is, the above-mentioned first image can include two eye images.

[0041] Optionally, in an embodiment of the present application, when a user uses the XR device 20, the infrared camera of the camera module 21 can capture the infrared signal of the user's hand in real time, the color image camera can capture the color image of the user's hand in real time, and the depth camera can capture the depth image of the user's hand in real time. By combining the infrared signal, color image and depth image of the user's hand, an image of the user's hand can be obtained, that is, a first image.

[0042] Optionally, in an embodiment of the present application, when a user uses the XR device 20, the camera module 21 may capture eye images and hand images of the user to obtain a first image, and the first image may include two eye images and one hand image.

[0043] It should be noted that the above description is only based on the example that the camera module 21 includes cameras for collecting eye images and hand images, and the camera module 21 collects eye images and hand images as the first image to illustrate the collection of the first image by the camera module 21, and does not constitute a limitation on the embodiments of the present application. Optionally, in the embodiments of the present application, the first image can be any image including an image of an interactive body, such as an eye image, a hand image, a facial image, etc.

[0044] Optionally, in the embodiment of the present application, the image of the interactive body may be an image obtained by imaging the interactive body through a camera, and the interactive body may be an object that interacts with the XR device 20. Exemplarily, the interactive body may be at least one of a hand, an eye, a face, a touch object, and a cursor. For example, the touch object may be a stylus, a touch stick, etc., and the cursor may be a light spot irradiated on the interface by a laser pen.

[0045] It should be noted that the first image may be a camera sensor image in RAW format, YUV format or other formats. Each pixel in the camera sensor image in RAW format is represented by four RGGB calculation units, and each pixel in the camera sensor image in YUV format is represented by three YUV calculation units. It can be seen that each pixel in the first image may be represented by at least three calculation units, and the amount of data is relatively large.

[0046] In the embodiment of the present application, the camera module 21 may further have a built-in cutout algorithm. After the camera module 21 captures the first image, the cutout algorithm may be used to extract the second image from the first image.

[0047] Optionally, in an embodiment of the present application, the camera module 21 is specifically used to identify the first image, and extract the image of the interactive body from the first image to obtain the second image.

[0048] In an embodiment of the present application, the camera module 21 can first identify whether the first image includes an image of the interactive body through a cutout algorithm. If so, determine the position information of the image of the interactive body in the first image, and extract the image of the interactive body from the first image according to the position information to obtain a second image.

[0049] Optionally, in an embodiment of the present application, the camera module 21 can first convert the first image into a grayscale image through a cutout algorithm, and perform noise reduction processing such as Gaussian blur and median filtering on the first image, and then identify the edge of the interactive body in the first image through edge detection technology, that is, the position information of the image of the interactive body in the first image, and then use image segmentation technology to separate the interactive body image from the background of the first image based on the position information of the image of the interactive body in the first image, and convert the separated interactive body image into a color image to obtain a second image.

[0050] In this way, by integrating the cutout algorithm in the camera module 21, before the camera module 21 sends the image to the image signal processor 221, the image of the interactive body, that is, the second image, is extracted from the first image, which can reduce the amount of data of the transmitted image, increase the transmission rate, and reduce the transmission power consumption.

[0051] Optionally, in an embodiment of the present application, the first image may be at least two images, and the at least two images may include an eye image and a hand image. In this case, the camera module 21 is specifically used to recognize the eye image, extract the eye image from the eye image, recognize the hand image, extract the gesture image from the hand image, and obtain the second image.

[0052] In the embodiment of the present application, when the first image includes an eye image, the interactive body may be an eye, and when the first image includes a hand image, the interactive body may be a hand. When the camera module 21 captures the eye image and the hand image, the camera module 21 may extract the eye image from the eye image and the gesture image from the hand image through a cutout algorithm.

[0053] Optionally, in the embodiment of the present application, taking the first image as an eye image as an example, the camera module 21 can first convert the eye image into a grayscale image through a cutout algorithm, and then convert the grayscale image into a binary image, that is, each pixel in the image is black (0) or white (255). Then, edge detection technology is used to identify the outline of the eye, eyelashes, pupil, etc., and then a clearer eye outline is obtained through morphological operations, and then machine learning or deep learning algorithms are used to extract eye features, such as the shape, size, position, etc. of the eye, and then template matching, sliding windows, etc. are used to locate the position of the eye image in the eye image based on the extracted features, and then the eye image is extracted from the eye image based on the position.

[0054] Optionally, in an embodiment of the present application, taking the first image as a hand image as an example, the camera module 21 can first perform target detection on the hand image based on a deep learning hand detection model through a cutout algorithm, identify the hand area, and determine the hand contour, and then use image segmentation technology to extract a gesture image from the hand image according to the hand contour.

[0055] In this way, a cutout algorithm is integrated in the camera module 21, which can extract an eye image from an eye image when an eye image is collected, and extract a gesture image from a hand image when a hand image is collected, thereby reducing the size of the image to be transmitted and reducing the time consumption of image transmission.

[0056] Optionally, in an embodiment of the present application, the camera module 21 is specifically used to generate a metadata table corresponding to the first image according to position information of the second image in the first image.

[0057] In the embodiment of the present application, the above-mentioned metadata table can be used to indicate the validity of a pixel unit in the first image, and the validity may include valid or invalid, and the pixel unit may include one or more pixel points.

[0058] It should be noted that, if the validity of a pixel unit is valid, it means that the pixel unit is a pixel unit in the image of the interactive body in the first image, that is, the pixel unit includes a part of the image of the interactive body; if the validity of a pixel unit is invalid, it means that the pixel unit is a pixel unit included in the image area other than the image of the interactive body in the first image, that is, the pixel unit does not include the image of the interactive body.

[0059] Optionally, in an embodiment of the present application, the distribution of cells in the above-mentioned metadata table is the same as the distribution of pixel units in the first image, and one cell in the metadata table corresponds to one pixel unit in the first image. Then, the value of each cell in the metadata table is used to indicate the validity of the pixel unit corresponding to the cell.

[0060] Optionally, in an embodiment of the present application, the above metadata table may be referred to as a meta table.

[0061] Optionally, in an embodiment of the present application, the above-mentioned camera module 21 is specifically used to construct an initial data table corresponding to the first image based on the distribution information of the pixel units in the first image; and fill in the initial data table according to the position information of the second image in the first image to obtain the metadata table corresponding to the first image.

[0062] Optionally, in an embodiment of the present application, the above-mentioned initial data table can be understood as a blank data table without filling in values. The data table can be a meta table or other types of data tables, and the embodiment of the present application does not limit this.

[0063] Optionally, in an embodiment of the present application, the distribution information of the pixel units in the first image may be understood as the number of pixel units in the horizontal direction and the number of pixel units in the vertical direction of the first image.

[0064] Optionally, in an embodiment of the present application, the camera module 21 may determine distribution information of pixel units in the first image according to a resolution of the first image.

[0065] In an example of the present application, the camera module 21 may determine the resolution of the first image as distribution information of pixel units in the first image, and a pixel unit may include a pixel point.

[0066] For example, assuming that the first image is an image with a resolution of 1920×1080, the distribution information of the pixel units in the first image can be 1920×1080, indicating that the first image has 1920 pixel units in the horizontal direction and 1080 pixel units in the vertical direction, and each pixel unit includes a pixel point.

[0067] In another example of the present application, the camera module 21 can determine every 2N pixels in the first image as a pixel unit, and then determine the distribution information of the pixel units in the first image according to the resolution of the first image. Wherein, N is an integer greater than 1, and the value of N can be set or adjusted according to the needs, and the embodiment of the present application does not limit this.

[0068] For example, assuming that the first image is an image with a resolution of 1920×1080, and taking the value of N as 2, the camera module 21 can determine every 4 pixels in the first image as a pixel unit, then according to the resolution of the first image, the distribution information of the pixel units in the first image is determined to be 960×540, indicating that the first image has 960 pixel units in the horizontal direction and 540 pixel units in the vertical direction, and each pixel unit includes 4 pixels.

[0069] Optionally, in an embodiment of the present application, the camera module 21 may construct the initial data table corresponding to the first image based on the distribution information of the pixel units in the first image, and the specific implementation may include: the camera module 21 may construct the initial data table corresponding to the first image with the number of pixel units in the horizontal direction of the first image as the number of rows and the number of pixel units in the vertical direction of the first image as the number of columns.

[0070] For example, assuming that the distribution information of the pixel units in the first image is 1920×1080, the number of rows and columns of the initial data table corresponding to the first image constructed by the camera module 21 is 1920, and the number of cells in the initial data table is 1920×1080.

[0071] Optionally, in an embodiment of the present application, the above-mentioned camera module 21 is specifically used to fill the initial data table by setting the value of the first cell in the initial data table to the first value and setting the value of the second cell in the initial data table to the second value according to the position information of the second image in the first image.

[0072] In the embodiment of the present application, the first cell is a cell corresponding to the pixel cell position in the second image, and the second cell is a cell other than the first cell in the initial data table. The first value is used to indicate that the pixel cell corresponding to the first cell is valid, and the second value is used to indicate that the pixel cell corresponding to the second cell is invalid.

[0073] Optionally, in an embodiment of the present application, after constructing the initial data table corresponding to the first image, the camera module 21 can determine the first cell in the initial data table according to the position information of the second image in the first image, set the value of the first cell to the first value, and then set the value of the second cell in the initial data table except the first cell to the second value to fill the initial data table corresponding to the first image and obtain the metadata table corresponding to the first image.

[0074] It should be noted that the first value and the second value are different values, that is, in the metadata table, different values ​​can be used to represent the validity of the pixel unit. For example, the first value can be set to 1 and the second value can be set to 0; or the first value can be set to 0 and the second value can be set to 1.

[0075] Optionally, in an embodiment of the present application, when the first image includes an eye image and a hand image, the camera module 21 can construct an initial data table corresponding to the eye image based on the distribution information of the pixel units in the eye image, and fill the initial data table by setting the value of the first cell in the initial data table to the first value and the value of the second cell in the initial data table to the second value according to the position of the eye image in the eye image, thereby obtaining a metadata table corresponding to the eye image. Similarly, the camera module 21 can construct an initial data table corresponding to the hand image based on the distribution information of the pixel units in the hand image, and fill the initial data table by setting the value of the first cell in the initial data table to the first value and the value of the second cell in the initial data table to the second value according to the position of the gesture image in the hand image, thereby obtaining a metadata table corresponding to the hand image.

[0076] It should be noted that the specific implementation of constructing the metadata table corresponding to the eye image and constructing the metadata table corresponding to the hand image can refer to the relevant description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0077] For example, if Figure 3As shown, assuming that the first image is an image in RGGB format, including 24 pixel units, and the first image includes an eye or a hand, the camera module 21 recognizes the first image through a cutout algorithm, and can determine that the position information of the second image (eye image or gesture image) in the first image is pixel 9, pixel 10, pixel 15, and pixel 16, that is, pixel 9, pixel 10, pixel 15, and pixel 16 are valid. Then, as Figure 4 As shown, the camera module 21 can construct a metadata table corresponding to the first image, each cell in the metadata table corresponds to a pixel unit in the first image, and the four cells with values ​​1 in the metadata table correspond to pixels 9, 10, 15 and 16 in the first image respectively, and the cells with values ​​0 are the areas in the first image excluding the second image.

[0078] In this way, the camera module 21 constructs a metadata table corresponding to the first image, and each cell in the metadata table corresponds to a pixel unit in the first image. The validity of the pixel unit in the metadata table can be determined according to the value of each cell in the metadata table, that is, the position of the image of the interactive body in the first image can be determined according to the metadata table. Therefore, it is not necessary to send the first image to the image signal processor 221. Sending the metadata table and the second image can also determine the user's input to the XR device 20, and can also reduce the amount of data sent by the camera module 21 to the image signal processor 221, reduce the time consumption of data transmission, and reduce the power consumption of data transmission. In addition, since the first image is an unprocessed camera sensor image, one pixel is represented by at least three values, while one pixel unit in the metadata table is represented by one value, which significantly reduces the data size of the first image, and can significantly reduce the amount of data transmitted, thereby reducing the time consumption of image transmission.

[0079] Optionally, in the embodiment of the present application, after the camera module 21 extracts the second image from the first image and generates a metadata table corresponding to the first image, the second image and the metadata table can be sent to the image signal processor 221. The above-mentioned image signal processor 221 is used to send the metadata table and the second image sent by the camera module 21 to the central processor 222.

[0080] Exemplarily, when the second image is an image in YUV format or RGB format, after receiving the metadata table and the second image sent by the camera module 21, the image signal processor 221 can directly send the metadata table and the second image to the central processor 222.

[0081] As another example, in the case where the second image is an image in RAW format, the image signal processor 221 may convert the second image from RAW format to YUV format or RGB format, and then send the metadata table and the second image in YUV format to the central processor 222, or send the metadata table and the second image in RGB format to the central processor 222.

[0082] Optionally, in an embodiment of the present application, after the camera module 21 extracts the second image from the first image and generates a metadata table corresponding to the first image, it can encapsulate the second image and the metadata table into a data packet in a first data format according to the MIPI interface standard and the communication protocol between the camera module 21 and the image signal processor 221.

[0083] Exemplarily, the first data format may be any one of a RAW format, an RGB format, a YUV format, and the like.

[0084] For example, if Figure 5 As shown, after the camera module 21 collects the original RAW data (the first image mentioned above), it determines and extracts the gaze point data (the second image mentioned above) through a cutout algorithm. The gaze point data includes eyes or hands, and the camera module 21 can count the valid data (pixel units in the second image) and invalid data (pixel units in the first image except the second image) in the original RAW data, and construct a meta table (metadata table), and then encapsulate the meta table and the gaze point data to obtain a data packet.

[0085] Optionally, in an embodiment of the present application, after the camera module 21 encapsulates the second image and the metadata table into a data packet in the first data format, it can send the data packet in the first data format to the image signal processor 221, and the image signal processor 221 is used to receive the data packet in the first data format.

[0086] Exemplarily, the camera module 21 may send a data packet in a first data format to the image signal processor 221 via a MIPI interface.

[0087] Optionally, in the embodiment of the present application, when the first data format is in YUV format or RGB format, after receiving a data packet in the first data format, the image signal processor 221 may send a data packet in the first data format to the central processor 222 .

[0088] Exemplarily, when the first data format is the YUV format or the RGB format, the image signal processor 221 may not adjust the data format of the data packet, and directly send the data packet in the first data format to the central processor 222 .

[0089] Optionally, in an embodiment of the present application, the above-mentioned image signal processor 221 is used to convert the metadata table and the second image sent by the camera module 21 from a first data format to a second data format, and send the metadata table and the second image in the second data format to the central processor 222.

[0090] Optionally, in an embodiment of the present application, the first data format may be a RAW format, and the second data format may be a YUV format or an RGB format or other data format that is convenient for the central processing unit 222 to process.

[0091] Exemplarily, when the first data format is the RAW format, after the image signal processor 221 receives a data packet in the first data format sent by the camera module 21, it can convert the data packet in the first data format into a second data format such as the YUV format or the RGB format, and then send the data packet in the second data format to the central processor 222.

[0092] In this way, the image signal processor 221 converts the received second image and metadata table in the first data format into the second data format, so that the data sent to the central processor 222 is in a data format that is easy to process, thereby improving data processing efficiency.

[0093] Optionally, in an embodiment of the present application, the central processing unit 222 may be a CPU, or may be a processor including a CPU, a graphics processing unit (GPU), a neural network processor (NPU), etc. that integrates various processing functions.

[0094] Optionally, in the embodiment of the present application, the central processor 222 is configured to perform corresponding operations based on the metadata table and the second image sent by the image signal processor 221 .

[0095] Optionally, in the embodiment of the present application, the above corresponding operation can be understood as an operation corresponding to the first image. For example, if the first image is a hand image, the corresponding operation can be an operation corresponding to a gesture in the hand image; if the first image is an eye image, the corresponding operation can be an operation on the position where the eye is looking in the eye image; if the first image includes a hand image and an eye image, the corresponding operation can be an operation corresponding to a gesture performed on the position where the eye is looking.

[0096] It should be noted that the first image in the embodiment of the present application may be a complete image captured by the XR device 20, and the first image may include an image of the interactive body, and may also include images of other objects around the interactive body. Exemplarily, the first image is a hand image, and the hand image may include an image of the hand, and may also include images of other objects around the hand. The first image is an eye image, and the eye image may include an image of the eye, and may also include images of other objects around the eye.

[0097] It should be noted that the second image in the embodiment of the present application may be a partial image extracted from the first image by the XR device 20, and the second image may include only the interactive body image. Exemplarily, the second image is a gesture image, and the gesture image may include only the hand image extracted from the hand image, and the second image is an eye image, and the eye image may include only the eye image extracted from the eye image.

[0098] Optionally, in an embodiment of the present application, the above-mentioned central processor 222 is specifically used to determine the interactive input to the XR device 20 based on the metadata table and the second image sent by the image signal processor 221; and in response to the interactive input, perform an operation corresponding to the interactive input.

[0099] Optionally, in the embodiment of the present application, the central processor 222 may have a built-in perception algorithm and an execution algorithm. The central processor 222 may be used to determine the user's interactive input to the XR device 20 according to the metadata table and the second image through the perception algorithm, and perform an operation corresponding to the interactive input through the execution algorithm. Exemplarily, the perception algorithm may be a hand-eye algorithm.

[0100] Optionally, in an embodiment of the present application, when the second image is two eye images, the central processor 222 may determine the position of the user's gaze point in the interface displayed by the XR device 20 based on a left eye image and a metadata table and a right eye image and a metadata table, and further determine that the user's interactive input is an input to the position of the gaze point.

[0101] Optionally, in an embodiment of the present application, when the second image is a gesture image, the above-mentioned central processor 222 can determine the position and gesture of the user's operation point in the interface displayed by the XR device 20 based on a gesture image and a metadata table, and then determine that the user's interactive input is the input corresponding to the gesture performed at the position of the operation point.

[0102] Optionally, in an embodiment of the present application, when the second image includes two eye images and a gesture image, the above-mentioned central processor 222 can determine the position of the user's gaze point in the interface displayed by the XR device 20 based on a left eye image and a metadata table and a right eye image and a metadata table, and determine the position of the user's operation point and gesture in the interface displayed by the XR device 20 based on a gesture image and a metadata table. When the position of the operation point and the position of the gaze point are the same, the position of the gaze point is determined to be the position of the user's interactive input. When the position of the operation point and the position of the gaze point are different, the position of the operation point can be determined as the position of the user's interactive input, and then it can be determined that the user's interactive input is the input corresponding to the gesture performed at the position.

[0103] Exemplarily, assuming that the CPU 222 determines that the user's gesture is a finger-clicking gesture on the interface, and the finger clicks at position A, then it can be determined that the user's interactive input is a click input or a short press input at position A.

[0104] It should be noted that the position and gesture of the user's interactive input at a certain moment can be determined through the above method, but the user's interactive input may be dynamic input, such as sliding, pinching, long pressing, double-clicking, drawing graphics, etc., which cannot be accurately judged only by the image at a certain moment. Therefore, the central processor 222 can determine the user's interactive input based on multiple frames of second images and multiple metadata tables.

[0105] Optionally, in the embodiment of the present application, the central processor 222 can determine the user's interactive input to the XR device 20 according to the multiple frames of second images and the multiple metadata tables through a perception algorithm. The multiple frames of second images correspond to the multiple metadata tables one by one, and each frame of the second image corresponds to a metadata table.

[0106] In an example of the present application, the second image is an eye image, and the central processor 222 can determine the position of the user's gaze point in the interface displayed by the XR device 20 based on multiple frames of eye images and multiple metadata tables, and then determine the user's interactive input.

[0107] For example, taking 10 frames of eye images as an example, assuming that the central processor 222 determines that the user's eyes are always looking at position A in the interface based on the 10 frames of eye images and 10 metadata tables, then the position of the gaze point can be determined to be position A, and further, it can be determined that the user's interactive input is a single-click input to position A.

[0108] In another example of the present application, the second image is a gesture image, and the central processor 222 can determine the position and gesture of the user's operation in the interface displayed by the XR device 20 based on multiple frames of gesture images and multiple metadata tables, and then determine the user's interactive input.

[0109] Example 1: Assume that in 10 frames of gesture images, the user's gestures are all gestures of clicking the interface with one finger. The first frame of gesture image is determined to be at position A of the first frame of the first image based on the first metadata table. Similarly, it can be determined that the position of each subsequent frame of gesture image in each frame of the first image gradually moves to the right. The tenth frame of gesture image is at position B of the tenth frame of the first image. It can be determined that the interactive input is a sliding input from position A to position B.

[0110] Example 2: Assuming that in the 20 frames of gesture images, the user's gestures are all gestures of clicking the interface with one finger, and according to the metadata table, it is determined that the position of each frame of gesture image in each frame of the first image is the same, that is, position C, then it can be determined that the interactive input is a long press input on position C.

[0111] Example 3: Assuming that in two frames of gesture images, the user's gestures are all gestures of clicking the interface with one finger, and according to the metadata table, it is determined that the position of each frame of gesture image in each frame of the first image is the same, that is, position C, then it can be determined that the interactive input is a short press input on position C.

[0112] Example 4: Assume that in 20 frames of gesture images, the first frame of gesture image shows two fingers separated, and the area where the two fingers are located is area D. The subsequent 19 frames of gesture images show the two fingers gradually closing, until the 19th frame of image shows the two fingers closed together. Then, it can be determined that the interactive input is a reduction input to area D.

[0113] Example 5: Assume that in 20 frames of gesture images, the first frame of gesture image is two fingers closed, and the area where the two fingers are located is area D. The subsequent 19 frames of gesture images are a process of the two fingers gradually separating, until the 19th frame of image is the two fingers separated and a certain distance apart. Then, it can be determined that the interactive input is an enlarged input to area D.

[0114] In another example of the present application, the second image includes an eye image and a gesture image. The central processor 222 can determine the position and gesture of the user's operation in the interface displayed by the XR device 20 based on multiple frames of gesture images and multiple metadata tables and multiple frames of eye images and multiple metadata tables, and further determine the user's interactive input.

[0115] Exemplarily, assuming that in 5 frames of gesture images, the user's gestures are all "OK" gestures, and the user's eyes are looking at the current interface, it can be determined that the interactive input is a confirmation input for the current interface.

[0116] Optionally, in the embodiment of the present application, the XR device 20 may further include a display screen. The central processor 222 is further configured to send a display instruction to the display screen, the display instruction being used to instruct the display screen to display an interface corresponding to the operation. The display screen is configured to display an interface corresponding to the operation in response to the display instruction sent by the central processor 222.

[0117] Optionally, in the embodiment of the present application, the display screen may be two small screens, one for each eye, and the display screen uses head-mounted display technology to project images onto the user's eyes, providing the user with an immersive virtual experience. Alternatively, the display screen may be a transparent display or transparent screen with special materials and structures, which uses transparent display technology to allow light to pass through from the back and present a virtual image in front, so that the user can see the real world and the virtual image at the same time.

[0118] Optionally, in the embodiment of the present application, after executing the operation corresponding to the interactive input, the CPU 222 can render the interface to be displayed after executing the operation, generate a display instruction, and send the display instruction to the display screen. After the display screen receives the display instruction, it can obtain the rendered interface and display it. Exemplarily, the interface corresponding to the operation displayed on the display screen can be a virtual interface.

[0119] In this way, the central processor 222 determines the user's interactive input based on the second image and the metadata table. Since the amount of data in the second image and the metadata table is less than that of the first image, the processing efficiency of the central processor 222 can be improved. Then the efficiency of the XR device 20 in executing the operations corresponding to the interactive input and displaying the corresponding interface will also be improved, which can bring a better user experience to the user.

[0120] Furthermore, in the process of using the XR device 20, due to the introduction of the hand-eye interaction function, the hand-eye interaction requires the camera module 21 to send the second image to the image signal processor 221 in the SOC 22 in real time, and the second image needs to have good smoothness. Therefore, the smoothness of the second image can be improved by increasing the frame rate of the first image, thereby improving the smoothness of the image displayed on the display screen.

[0121] Next, combine Figure 6 After the frame rate is increased from 60fps to 90fps, the time consumed in transmitting data from the camera module 21 to the image signal processor 221 is described. Figure 6 The figure shows the transmission time of the nth frame and the n+1th frame data. Figure 6It can be seen that the time taken to transmit data through the MIPI interface, namely the Output Data transmission time (MIPI transmission time), is fixed at 8ms. When the frame rate is 60fps, the duration of each frame is 16.6ms, and when the frame rate is 90fps, the duration of each frame is 11.1ms. Therefore, after increasing the frame rate, the adjustable range of the exposure time (Exposure) is significantly reduced, from 0-8ms to 0-3ms. In addition, the duration of each frame also includes the blanking duration and the delay duration (Read Delay).

[0122] In addition, when the ambient brightness is low, in order to improve the image quality, for the XR device 20 equipped with an infrared LED fill light, the infrared LED fill light can be turned on for fill light. For the XR device 20 not equipped with an infrared LED fill light, the image brightness can be improved by increasing the exposure time.

[0123] In the embodiment of the present application, since the cutout algorithm is built into the camera module 21, the amount of data sent by the camera module 21 to the image signal processor 221 is reduced, and the data transmission time, i.e., the Output Data transmission time, is shortened. Therefore, when using this solution, in order to improve the smoothness of the image, even if the frame rate of the image is increased, resulting in a smaller adjustable range of the exposure time, the adjustable range will not be very small. In this case, if fill light is required, the image brightness can be improved by adjusting the exposure time, and there is no need to add an infrared LED lamp, or even if an infrared LED lamp is required, since a longer exposure time can increase a part of the fill light effect, the duration of turning on the LED lamp will be relatively short. Compared with the related art, the adjustable range of the exposure time is very small, resulting in the need to turn on the LED lamp for a long time. This solution can improve the brightness of the image while improving the smoothness of the image, and reduce the power consumption of the transmitted data and the power consumption of the infrared LED lamp. In addition, the solution of the present application increases the adjustable range of the exposure time, so the XR device 20 can also be used when the ambient brightness is low, thereby increasing the flexibility of adjusting the exposure time.

[0124] It can be seen that when using the XR device 20 provided in the embodiment of the present application, the adjustable range of exposure time is significantly larger than the exposure range that can be adjusted by the current camera frame rate.

[0125] The XR device provided in the embodiment of the present application has a cutout algorithm built into the camera module. After the camera module captures the first image, it does not send the first image directly to the image signal processor, but sends the second image extracted from the first image and the generated metadata table to the image signal processor. Since the second image is extracted from the first image, the second image is smaller than the first image, and the metadata table uses one data to represent a pixel unit in the first image. Compared with using three-dimensional or four-dimensional data to represent a pixel unit in the first image, the amount of data is greatly reduced. It can be seen that the sum of the data amounts of the second image and the metadata table is smaller than that of the first image. Therefore, the amount of data transmitted from the camera module to the image signal processor is reduced, so the time consumption of the transmission process will also be reduced, the resources required for transmitting data will also be reduced, and the power consumption of transmitting data will also be reduced. In addition, the above method can improve the frame rate of the image, thereby improving the smoothness of the image. When the frame rate is increased, the power consumption of the XR device's data transmission bandwidth and the power consumption of the infrared LED lamp can be reduced, the time taken to transmit data from the camera module to the SOC can be reduced, and the real-time nature of the image display can be increased. The above method also increases the adjustable range of the exposure time, thereby improving the flexibility of the XR device's usage scenarios.

[0126] The XR device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0127] The present application also provides an image processing method, which can be applied to scenarios where users use interactive functions of XR devices. For example, the above scenario can be specifically that users use XR devices to play games, watch movies, shop, etc., which is not limited in the present application.

[0128] In some embodiments of the present application, the execution subject of the above-mentioned image processing method may be the above-mentioned XR device. The following will take the execution subject as an XR device as an example to exemplify the image processing method provided in the embodiments of the present application.

[0129] Figure 7 is a flow chart of the image processing method provided in the embodiment of the present application, such as Figure 7 As shown, the image processing method provided in the embodiment of the present application may include the following steps 701 to 706.

[0130] Step 701: The XR device captures a first image through a camera module.

[0131] Step 702: The XR device extracts the second image from the first image through the camera module, and generates a metadata table corresponding to the first image.

[0132] Optionally, in an embodiment of the present application, in the above step 702, "the XR device extracts the second image from the first image through the camera module" can be specifically implemented through the following step 7021.

[0133] Step 7021: The XR device recognizes the first image through the camera module, and extracts the image of the interactive object from the first image to obtain the second image.

[0134] In the embodiment of the present application, the above-mentioned interactive body may be an object that interacts with the XR device.

[0135] Optionally, in an embodiment of the present application, the first image may be at least two images, and the at least two images may include an eye image and a hand image. In the above step 702, "the XR device extracts the second image from the first image through the camera module" may be specifically implemented by the following step 7022.

[0136] Step 7022: The XR device recognizes the eye image through the camera module, extracts the eye image from the eye image, recognizes the hand image, extracts the gesture image from the hand image, and obtains the second image.

[0137] Optionally, in the embodiment of the present application, the above step 702 can be implemented through the following step 7023.

[0138] Step 7023: The XR device generates a metadata table corresponding to the first image through the camera module according to the position information of the second image in the first image.

[0139] Optionally, in an embodiment of the present application, the above step 702 can be implemented by the following steps 7024 and 7025.

[0140] Step 7024: The XR device constructs an initial data table corresponding to the first image through the camera module according to the distribution information of the pixel units in the first image.

[0141] Step 7025: The XR device fills the initial data table through the camera module according to the position information of the second image in the first image, and obtains the metadata table corresponding to the first image.

[0142] Optionally, in an embodiment of the present application, the above step 7025 can be implemented by the following step 70251.

[0143] Step 70251, the XR device fills the initial data table through the camera module by setting the value of the first cell in the initial data table to the first value and setting the value of the second cell in the initial data table to the second value according to the position information of the second image in the first image.

[0144] In the embodiment of the present application, the first cell is a cell corresponding to the pixel cell position in the second image, and the second cell is a cell other than the first cell in the initial data table. The first value is used to indicate that the pixel cell corresponding to the first cell is valid, and the second value is used to indicate that the pixel cell corresponding to the second cell is invalid.

[0145] Step 703: The XR device sends the metadata table and the second image to the image signal processor through the camera module.

[0146] In the embodiment of the present application, the above metadata table is used to indicate the validity of the pixel unit in the first image.

[0147] Step 704: The XR device receives the metadata table and the second image through an image signal processor.

[0148] Step 705: The XR device sends the metadata table and the second image to the central processor via the image signal processor.

[0149] Optionally, in the embodiment of the present application, the above step 705 can be specifically implemented through the following step 7051.

[0150] Step 7051: The XR device converts the metadata table and the second image from the first data format to the second data format through the image signal processor, and sends the metadata table and the second image in the second data format to the central processor.

[0151] In the embodiment of the present application, the first data format may be a RAW format, and the second data format may be a YUV format or an RGB format.

[0152] Step 706: The XR device performs corresponding operations based on the metadata table in the second data format and the second image through the central processor.

[0153] Optionally, in the embodiment of the present application, the above step 706 can be specifically implemented through the following step 7061.

[0154] Step 7061: The XR device determines the interactive input to the XR device through the central processor based on the metadata table and the second image sent by the image signal processor, and performs an operation corresponding to the interactive input in response to the interactive input.

[0155] Optionally, in the embodiment of the present application, the XR device may further include a display screen. After the above step 7061, the image processing method provided in the embodiment of the present application may further include the following steps 7062 and 7063.

[0156] Step 7062: The XR device sends a display instruction to the display screen through the central processor, where the display instruction is used to instruct the display screen to display an interface corresponding to the operation.

[0157] Step 7063: The XR device displays an interface corresponding to the operation through the display screen in response to the display instruction sent by the central processor.

[0158] The image processing method provided in the embodiment of the present application, after the camera module captures the first image, does not directly send the first image to the image signal processor, but sends the second image extracted from the first image and the generated metadata table to the image signal processor. Since the second image is extracted from the first image, the second image is smaller than the first image, and the metadata table uses one data to represent one pixel unit in the first image. Compared with the first image, which uses three-dimensional or four-dimensional data to represent one pixel unit, the amount of data is greatly reduced. It can be seen that the sum of the amount of data of the second image and the metadata table is smaller than the first image. Therefore, the amount of data transmitted from the camera module to the image signal processor is reduced, so the time consumption of the transmission process will also be reduced, the resources required for data transmission will also be reduced, and the power consumption of data transmission will also be reduced. In addition, the frame rate of the image can be improved by the above method, thereby improving the smoothness of the image, and when the frame rate is increased, the power consumption of the data bandwidth of the XR device and the power consumption of the infrared LED lamp can also be reduced, the time consumption of data transmission from the camera module to the SOC is reduced, and the real-time performance of the image display is increased. In addition, the above method increases the adjustable range of the exposure time and improves the flexibility of the use scenario of the XR device.

[0159] It should be noted that the specific implementation process of the above steps 701 to 706 can refer to the relevant description of the above XR device embodiment. To avoid repetition, this embodiment will not be repeated here.

[0160] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not impose any restrictions on this.

[0161] The XR device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be repeated here.

[0162] Those skilled in the art will appreciate that the XR device in the embodiment of the present application may also include Figure 2 More or fewer components, or combinations of certain components, or different arrangements of components are not described in detail here.

[0163] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0165] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An XR device, characterized in that: The XR device includes a camera module and a system on chip (SOC), and the SOC includes an image signal processor and a central processing unit; The camera module is used to capture a first image, extract a second image from the first image, generate a metadata table corresponding to the first image, and send the metadata table and the second image to the image signal processor, wherein the metadata table is used to indicate the validity of a pixel unit in the first image; The image signal processor is used to send the metadata table and the second image sent by the camera module to the central processor; The central processor is used to perform corresponding operations based on the metadata table and the second image sent by the image signal processor.

2. The XR device according to claim 1, characterized in that The camera module is specifically used to generate a metadata table corresponding to the first image according to the position information of the second image in the first image.

3. The XR device according to claim 2, characterized in that The camera module is specifically used to construct an initial data table corresponding to the first image according to the distribution information of the pixel units in the first image; and fill in the initial data table according to the position information of the second image in the first image to obtain the metadata table corresponding to the first image.

4. The XR device according to claim 3, characterized in that The camera module is specifically used to fill the initial data table by setting the value of the first cell in the initial data table to the first value and setting the value of the second cell in the initial data table to the second value according to the position information of the second image in the first image; Among them, the first cell is a cell corresponding to the pixel unit position in the second image, the second cell is a cell in the initial data table other than the first cell, the first value is used to represent that the pixel unit corresponding to the first cell is valid, and the second value is used to represent that the pixel unit corresponding to the second cell is invalid.

5. The XR device according to claim 1, characterized in that The camera module is specifically used to identify the first image and extract the image of the interactive object from the first image to obtain the second image, where the interactive object is an object that interacts with the XR device.

6. The XR device according to any one of claims 1 to 5, characterized in that: The first image is at least two images, and the at least two images include an eye image and a hand image; The camera module is specifically used to identify the eye image, extract the eye image from the eye image, and identify the hand image, extract the gesture image from the hand image to obtain the second image.

7. The XR device according to any one of claims 1 to 5, characterized in that The central processor is specifically configured to determine an interactive input to the XR device based on the metadata table and the second image sent by the image signal processor; and in response to the interactive input, perform an operation corresponding to the interactive input.

8. The XR device according to claim 7, characterized in that The XR device also includes a display screen; The central processor is further used to send a display instruction to the display screen, wherein the display instruction is used to instruct the display screen to display an interface corresponding to the operation; The display screen is used to display the interface corresponding to the operation in response to the display instruction sent by the central processing unit.

9. The XR device according to any one of claims 1 to 5, characterized in that The image signal processor is specifically used to convert the metadata table and the second image sent by the camera module from a first data format to a second data format, and send the metadata table and the second image in the second data format to the central processing unit.

10. The device according to claim 9, characterized in that The first data format is the original RAW format, and the second data format is the brightness and chromaticity YUV format or the red, green and blue RGB format.

Images