Head-mounted device, control method thereof and storage medium
By performing pixel merging and graying processing on color images in VR headsets to generate graying images, the problem of increasing weight and power consumption of multiple cameras in the prior art is solved, and the equipment is lightweight and energy efficiency improvement is achieved.
Patent Information
- Application Number
- CN202310556071.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-22
AI Technical Summary
The existing VR headsets are equipped with multiple cameras to increase the weight and power consumption of the device. How to reduce the number of cameras without affecting the functions has become a focus of the industry.
Reliance on color cameras is reduced by acquiring color images in a head-mounted device and performing pixel merging and grayscale processing, and producing grayscale images with lower resolution for machine vision applications.
This achieves a reduction in camera count, reduces the cost, weight and power consumption of the equipment, while meeting the needs of color perspective imaging and machine vision applications.
Smart Images

Figure CN119002050A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of head-mounted devices, and in particular to a head-mounted device and a control method and a storage medium thereof. Background Art
[0002] With the continuous development of VR (Virtual Reality) technology, VR headsets have begun to be popularized to more and more individual users, and people's requirements for VR headsets are getting higher and higher. At present, common VR headsets generally use at least one color camera and at least four black and white cameras to capture color images and grayscale images corresponding to the real environment to achieve the color perspective imaging (C-VST) function of the VR headset, as well as the positioning and posture recognition (such as 6DoF function) of the VR headset itself. However, carrying at least five cameras will increase the weight of the VR headset, and the power consumption of the VR headset will also be higher. Therefore, how to reduce the number of cameras carried by VR headsets without affecting existing functions has become the main focus of industry insiders. Summary of the invention
[0003] On the one hand, the present application provides a control method for a head-mounted device, the control method comprising: acquiring a color image; performing pixel merging and grayscale processing on the color image in a preset order to generate a first grayscale image; and executing a machine vision application based on the first grayscale image.
[0004] On the other hand, the present application provides a head-mounted device, which includes: a head-mounted display mechanism, and a camera and a processor arranged on the head-mounted display mechanism; the camera is configured to capture a color image, and the head-mounted display mechanism is configured to display the color image; the processor is configured to perform pixel merging and grayscale processing on the color image in a preset order to generate a first grayscale image, and execute a machine vision application based on the first grayscale image.
[0005] The present application also provides a computer-readable storage medium, in which program data is stored. When the program data is executed by a processor, it is used to implement the above-mentioned control method.
[0006] The control method of the head-mounted device provided in the present application performs pixel merging and grayscale processing on the acquired color image in a preset order, so that the color image can be converted into a first grayscale image with a resolution lower than the color image, so as to execute a machine vision application based on the first grayscale image. In this way, the head-mounted device is only equipped with a color camera that can acquire color images, and can also obtain grayscale images for executing machine vision applications, which is conducive to reducing the number of cameras and reducing the cost, weight and power consumption of the head-mounted device. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0008] Figure 1 is a schematic structural diagram of a head mounted device 10 provided in the present application;
[0009] Figure 2 yes Figure 1 Another structural schematic diagram of the head mounted device 10;
[0010] Figure 3 yes Figure 2 A schematic diagram of the structure of the head display mechanism 100 and the camera 200;
[0011] Figure 4 yes Figure 3 Another structural schematic diagram of the head display mechanism 100 and the camera 200;
[0012] Figure 5 is a schematic diagram of pixel arrangement of the image sensor 201 provided by the present application;
[0013] Figure 6 yes Figure 5 Another schematic diagram of pixel arrangement of the image sensor 201;
[0014] Figure 7 yes Figure 5 Another schematic diagram of pixel arrangement of the image sensor 201;
[0015] Figure 8 is a flowchart of a control method of a head mounted device 10 provided in the present application;
[0016] Fig. 9 It is a flowchart of data processing of the head mounted device 10 provided in the present application;
[0017] Fig.10 yes Fig. 9 Another schematic diagram of the process of data processing of the head mounted device 10;
[0018] Fig.11 This is an example of pixel merging for a color image;
[0019] Fig.12 yes Fig.11 Another example of pixel merging for a color image in FIG.
[0020] Fig.13 is another flowchart of the control method of the head mounted device 10 provided in the present application;
[0021] Fig.14 is another flowchart of data transmission of the head mounted device 10 provided in the present application;
[0022] Fig.15 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0023] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0024] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0025] The present application describes a head-mounted device. The head-mounted device can be a virtual reality device, and the head-mounted device can be configured to transfer data to an external processing device and receive data from the external processing device through a signal connection, and the signal connection can be a wired connection, a wireless connection, or a combination thereof. However, in other cases, the head-mounted device can be used as a stand-alone device, that is, data processing is performed on the head-mounted device itself. The signal connection can be configured to carry any kind of data, such as image data (e.g., still images and / or full motion video, including 2D and 3D images), audio, multimedia, voice, and / or any other type of data. The external processing device can be, for example, a game console, a personal computer, a tablet computer, a smart phone, or other types of processing devices. The signal connection can be, for example, a universal serial bus (USB) connection, a Wi-Fi connection, a Bluetooth or Bluetooth low energy (BLE) connection, an Ethernet connection, a cable connection, a DSL connection, a cellular connection (e.g., 3G, LTE / 4G or 5G), etc., or a combination thereof. Additionally, the external processing device may communicate with one or more other external processing devices via a network, which may be or include, for example, a local area network (LAN), a wide area network (WAN), an intranet, a metropolitan area network (MAN), the global Internet, or a combination thereof.
[0026] See also Figures 1 to 4 , Figure 1 is a schematic diagram of the structure of a head mounted device 10 provided in an embodiment of the present application, Figure 2 yes Figure 1 Another structural schematic diagram of the head mounted device 10 is shown in FIG. Figure 3 yes Figure 2 A schematic diagram of the structure of the head display mechanism 100 and the camera 200, Figure 4 yes Figure 3 Another structural schematic diagram of the head display mechanism 100 and the camera 200.
[0027] The head mounted device 10 provided in the embodiment of the present application can be worn on the head of a user and can realize the function of virtual reality in front of the user's eyes. Figure 1 to Figure 2 As shown, the head mounted device 10 may include: a head mounted display mechanism 100, a camera 200 and a processor 300. The head mounted display mechanism 100 may be worn on the user's head through a structure such as a strap or a wearing bracket, and realize the function of virtual reality in front of the user's eyes. The camera 200 is arranged on the head mounted display mechanism 100, and the camera 200 is used to capture a color image of the real environment, and output the color image to the head mounted display mechanism 100 for display, so as to realize the color perspective imaging function (image display application) of the head mounted device 10. At the same time, the color image of the real environment captured by the camera 200 can also be converted into a first grayscale image with a lower resolution after processing for internal call of the head mounted device 10, so as to execute the machine vision application of the head mounted device 10. The processor 300 is arranged in the head mounted display mechanism 100, and the processor 300 can be used to process the image data captured by the camera 200, and execute corresponding instructions based on the processed image data, so as to realize the function of the head mounted device 10. Through the above settings, the camera 200 only needs to capture the color image and can also obtain the first grayscale image in subsequent processing without using other devices independent of the camera 200 (such as a black and white camera) to capture the first grayscale image, which is beneficial to reducing the number of cameras 200.
[0028] It is understandable that the aforementioned image display application is mainly used to display images for users to observe, and has a direct interactive relationship with the user, so it has high requirements for the real image captured by the camera 200, especially in human visual sensitive parameters such as scene color and image resolution. The difference is that the machine vision application mainly includes internal functions of the head-mounted device 10 such as 6DoF tracking, gesture recognition and 3D measurement, and has no direct interactive relationship with the user. Therefore, the requirements for the real image captured by the camera 200 are different from those of the image display application, especially in parameters required for machine vision calculations such as signal-to-noise ratio and dynamic range. Based on this, this implementation uses the aforementioned color image and the first grayscale image as input data for the image display application and the machine vision application respectively according to the different focuses of the image display application and the machine vision application, so as to respectively execute the application of the head-mounted device 10, which is conducive to reducing the algorithm power consumption of the head-mounted device 10 in data processing.
[0029] The head display mechanism 100 has a display (not shown), and the display can be used to execute image display applications, that is, to display the image of the real environment captured by the camera 200, so as to realize the color perspective imaging function of the head-mounted device 10. Among them, the display can have a refresh rate of at least greater than 60Hz to ensure that the picture displayed by the display has a high degree of smoothness. For example, the refresh rate of the display can be 60Hz, 90Hz or 120Hz. At the same time, the display can also have a resolution of 2K to ensure that the picture displayed by the display has sufficient clarity. For example, the resolution of the display can be 2560×1440, 2560×1600 or 2048×1080, etc. Optionally, in addition to the above refresh rate and resolution, the refresh rate and resolution of the display can also be selected according to design requirements, which is not limited in this embodiment.
[0030] The head display mechanism 100 also has a mounting side 110 that is disposed opposite to the display, and when the user wears the head display mechanism 100, the mounting side 110 is also disposed away from the user's eyes for mounting the camera 200, so that the camera 200 can capture images of the real environment from an angle similar to the user's eyes. Among them, the mounting side 110 also has a top 111 and a bottom 112 that are disposed opposite to each other, and a side 113 that connects the top 111 and the bottom 112 respectively. At the same time, when the user wears the head display mechanism 100, the top 111 can be a part of the mounting side 110 close to the top of the user's head, and the bottom 112 can be a part of the mounting side 110 close to the user's body. Optionally, in addition to the display, the head display mechanism 100 can also be equipped with other functional devices required by the head-mounted device 10, such as an inertial measurement sensor (Inertial Measurement Unit, IMU), which is used to measure inertial data and cooperate with the aforementioned first grayscale image to perform machine vision applications, which is not limited in this embodiment.
[0031] The camera 200 is disposed on the mounting side 110 of the head display mechanism 100, and the camera 200 can be used to capture a color image of the real environment and transmit it to the display of the head display mechanism 100 for display, so as to realize the color perspective imaging function of the head mounted device 10. Figures 2 to 3 As shown, the number of cameras 200 may be two, and they are respectively a first camera 210 and a second camera 220. Among them, the first camera 210 and the second camera 220 are both arranged on the mounting side 110, and are symmetrically arranged about the center line Y of the mounting side 110, and the optical axis L1 of the first camera 210 and the optical axis L2 of the second camera 220 may also be parallel. At the same time, when the user wears the head display mechanism 100, the first camera 210 and the second camera 220 may be arranged respectively relative to the left and right eyes of the user, and respectively capture the real images corresponding to the left and right eyes of the user, so as to match the actual visual range and visual effect of the user's eyes as much as possible. For example, the center distance between the first camera 210 and the second camera 220 may be 63mm (typical value of the pupil distance of the human eye). In this embodiment, the real images captured by the first camera 210 and the second camera 220 can be transmitted to the display of the head display mechanism 100 after processing for display, so as to realize the color perspective imaging function of the head mounted device 10.
[0032] Optionally, the number of cameras 200 may be one, and the real image captured by a single camera 200 may be split into color images corresponding to the left and right eyes of the user and output to the display of the head display mechanism 100 for display. Similarly, the color image captured by a single camera 200 may also be converted into a first grayscale image after processing and called as input data of the machine vision application.
[0033] Optionally, the optical axis L1 of the first camera 210 and the optical axis L2 of the second camera 220 may also be arranged non-parallel to expand the field of view M of the first camera 210 and the second camera 220 as much as possible, thereby increasing the coverage of the field of view M of the first camera 210 and the second camera 220. Figure 4 As shown, the optical axis L1 of the first camera 210 can be tilted toward the side 113 to reduce the overlapping area of the field of view M of the first camera 210 and the second camera 220, so that the field of view M of the first camera 210 can expand outward toward the direction close to the side 113, thereby expanding the coverage range of the field of view M of the first camera 210 and the second camera 220.
[0034] Optionally, since the user's gesture action range is generally biased towards the bottom and side of the user's head, in order to facilitate the camera 200 to capture the user's gesture action to achieve machine vision applications such as gesture recognition, the optical axis L1 of the first camera 210 can also be tilted toward the direction close to the bottom 112, so that the coverage range of the field of view angle M of the first camera 210 and the second camera 220 is biased towards the bottom and side of the user's head. In this embodiment, the field of view angle M of the first camera 210 and the second camera 220 can be in the range of 150° to meet the field of view requirements of image display applications (not less than 96°) and machine vision applications (not less than 130°).
[0035] Optionally, in addition to the optical axis L1 of the first camera 210 being tilted, the optical axis L2 of the second camera 220 may also be tilted, and the specific tilting method may be the same or similar to the tilting method of the optical axis L1. That is, at least one of the optical axis L1 of the first camera 210 and the optical axis L2 of the second camera 220 may be tilted toward the direction close to the side 113, and at least one may also be tilted toward the direction close to the bottom 112, as long as the coverage of the field of view angle M of the first camera 210 and the second camera 220 can be expanded, and this embodiment is not limited to this.
[0036] Optionally, when the optical axis L1 of the first camera 210 and the optical axis L2 of the second camera 220 are tilted, in order to reduce the influence of the tilt of the optical axis L1 and the optical axis L2 on the imaging of the camera 200, the first camera 210 and the second camera 220 can establish a non-tilted virtual optical axis through an algorithm for correcting the imaging effect, so as to ensure the matching degree between the real image captured by the first camera 210 and the second camera 220 and the real environment. It can be understood that the above description of "camera", "first camera" and "second camera" only represents the difference in quantity. In some embodiments, "camera" can also be referred to as "first camera" or "second camera", and "first camera" can also be referred to as "second camera" and "camera", and "second camera" can also be referred to as "camera" and "first camera".
[0037] See also Figures 5 to 7 , Figure 5 is a schematic diagram of pixel arrangement of the image sensor 201 provided in the present application, Figure 6 yes Figure 5 Another schematic diagram of the pixel arrangement of the image sensor 201 is shown in FIG. Figure 7 yes Figure 5 FIG. 2 is another schematic diagram of the pixel arrangement of the image sensor 201 .
[0038] The exposure mode of the camera 200 is global exposure to meet the needs of some machine vision applications (such as 6DoF tracking). At the same time, the output frame rate of the camera 200 is at least greater than 60 frames per second to ensure the smoothness of the display screen. For example, the output frame rate of the camera 200 can be 60 frames, 90 frames, or 120 frames. In addition, the camera 200 can be used to receive primary color light in the real environment, that is, red light, blue light, and green light, to capture a color image that matches the real environment, thereby realizing the color perspective imaging function of the head-mounted device 10. In this embodiment, the camera 200 can have an image sensor 201 (such as a Bayer array) using a Bayer array. Figure 5 As shown), the camera 200 receives the primary color light in the real environment for imaging. With such a configuration, since the conventional Bayer array image sensor 201 is adopted, the image sensor of the camera 200 does not need to be specially customized, and can also match the mature ISP (Image Signal Processing) restoration algorithm, which is beneficial to improve the matching degree between the real image captured by the camera 200 and the real environment after being processed, and avoid the problem of color cast (such as yellow cast) of the real image captured by the camera 200 after being processed by the restoration algorithm.
[0039] Optionally, in addition to receiving primary color light in the real environment, the camera 200 can also simultaneously receive primary color light and infrared light in the real environment to capture an image formed by the primary color light and the infrared light, and the image can generate a color image and a second grayscale image after processing, so that the processor 300 can execute an image display application based on the color image and execute a machine vision application based on the second grayscale image. At the same time, the color image can also be converted into a first grayscale image after processing as in the aforementioned embodiment, so as to cooperate with the second grayscale image captured by the camera 200 as input data for the machine vision application for the head display mechanism 100 to call. In this embodiment, the camera 200 can have an image sensor 201 (such as an RGB-IR array) using a RGB-IR array. Figure 6 As shown in FIG. 1 , the image sensor 201 of the RGB-W array can also be used to simultaneously receive the primary color light and the infrared light in the real environment for imaging. Figure 7As shown in the figure, the primary color light and infrared light in the real environment are received for imaging. That is, the IR channel in the RGB-IR array is replaced by the W channel, and the W channel can pass the primary color light and infrared light in the real environment at the same time. The imaging principle of the RGB-IR array and the RGB-W array is specifically described in the following method embodiment, which will not be expanded here.
[0040] Understandably, Figure 6 and Figure 7 The pixel arrangement of the RGB-IR and RGB-W arrays is only for schematic illustration. In some optional embodiments, the pixel arrangement in the RGB-IR and RGB-W arrays can also be adaptively adjusted according to needs, which is not limited in this implementation.
[0041] The processor 300 may be disposed in the head display mechanism 100, and the processor 300 may respectively execute image display applications and machine vision applications based on color images and grayscale images. The processor 300 may also be referred to as a CPU (Central Processing Unit). The processor 300 may be an integrated circuit chip having signal processing capabilities. The processor 300 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor 300 may be a microprocessor or any conventional processor, etc. In addition, the number of processors 300 may also be multiple, so as to process different data separately, such as ISP processing, image rendering, image cropping, image reconstruction, sampling, etc.
[0042] Optionally, the head mounted device 10 may also be provided with a laser emitter 400 mounted on the head display mechanism 100. Figure 2As shown, the laser emitter 400 can be arranged on the mounting side 110 and between the first camera 210 and the second camera 220, and can be used to emit infrared light, and the infrared light can be received by the camera 200 after being reflected by an object in the real environment, so that the camera 200 can perform imaging by receiving the infrared light in the real environment. In this way, the laser emitter 400 can be used as a light source, and the camera 200 can receive the infrared light emitted by the laser emitter 400 into the real environment, so that the camera 200 can obtain the second grayscale image in the aforementioned embodiment for the processor 300 to perform the machine vision application. In this embodiment, the laser emitter 400 can specifically emit an infrared laser (850nm) with a pseudo-random pattern to add 3D structured light data (second grayscale image) to the head-mounted device 10, so as to achieve accurate measurement of 3D scene information in the effective field of view (ranging or depth measurement, etc.), which is helpful for rendering the correct occlusion relationship between virtual and real objects in the scene during color perspective imaging.
[0043] Optionally, the design of the laser emitter 400 may also be omitted, that is, the laser emitter 400 may not be provided on the head display mechanism 100, and the laser emitter 400 may be used by the user as a device independent of the head-mounted device 10. For example, the laser emitter 400 may be fixed in a certain scene (such as a room), and when the user needs the laser emitter 400 to be used in conjunction with the head-mounted device 10 in the scene, the laser emitter 400 receives a start command issued by the head-mounted device 10 to emit an infrared laser into the scene. Alternatively, the laser emitter 400 may also be a portable device held by the user, and the user may choose whether to carry the laser emitter 400 for use according to needs.
[0044] The head mounted device 10 provided in the embodiment of the present application can capture color images in a real environment by setting a camera 200, and can convert the color images into grayscale images for the processor 300 to execute machine vision applications. With such a setting, it is not necessary to independently set up other cameras (such as black and white cameras) for capturing grayscale images, which is conducive to reducing the number of cameras 200 and reducing the cost, weight and power consumption of the head mounted device 10.
[0045] See also Figures 8 to 10 , Figure 8 is a flowchart of a control method of a head mounted device 10 provided in the present application, Fig. 9 is a flowchart of data processing of the head mounted device 10 provided in this application, Fig.10 yes Fig. 9 Another flowchart of data processing of the head mounted device 10 is shown in FIG. 1. In this embodiment, the control method of the head mounted device 10 is performed by the head mounted device 10 in the above embodiment.
[0046] like Figure 8As shown, the control method includes the following steps:
[0047] S13, obtaining a color image.
[0048] S14, executing an image display application based on the color image.
[0049] S15, performing pixel merging and grayscale processing on the color image in a preset order to generate a first grayscale image.
[0050] S16, executing a machine vision application based on the first grayscale image.
[0051] Specifically, the color image can be directly captured by the camera 200. For example, the camera 200 can have an image sensor 201 with a Bayer array (e.g., Figure 5 As shown in FIG. 1 ), the camera 200 can receive the primary color light in the real environment for imaging to capture a color image corresponding to the real environment. At the same time, the color image can be used as input data for the image display application, so that the processor 300 can execute the image display application based on the color image to realize the color perspective imaging function of the head mounted device 10 (such as Figures 9 and 10 As shown). In this embodiment, the resolution of the color image can be 1600×1300, and the output frame rate of the camera 200 can be 60 frames per second, so as to ensure that the picture seen by the user has sufficient clarity and smoothness while reducing the delay of the image sensor 201 as much as possible (generally controlled within 20ms). Of course, the resolution of the color image and the output frame rate of the camera 200 can also be adaptively adjusted according to design requirements, and this embodiment is not limited to this.
[0052] Optionally, the color image may be obtained by subsequent processing of the real image captured by the camera 200, and is not limited to being directly captured by the camera 200. Optionally, step S14 may also be omitted. That is, the color image may also be not limited to being used as input data of the image display application to execute the image display application. Optionally, the order in which step S14 and step S15 are executed may not be limited. That is, the processor 300 may first execute any one of step S14 and step S15, and then execute the other of step S14 and step S15, which is not limited in this embodiment.
[0053] Furthermore, after the camera 200 captures a color image corresponding to the real environment, it can also perform pixel merging and grayscale processing on the color image in a preset order to generate a first grayscale image as input data for the machine vision application, so that the processor 300 can execute the machine vision application (such as Figures 9 and 10As shown). The aforementioned preset order refers to the order of pixel merging and grayscale processing, and the preset order can be saved in the memory after being set by the designer for the processor 300 to call and execute. For example, step S15 may include:
[0054] S151, performing pixel merging on the color image.
[0055] S152, grayscale processing is performed on the color image after pixel merging.
[0056] That is, after acquiring the color image, the color image can be first pixel-merged to reduce the resolution of the color image, and then the color image with reduced resolution can be grayed to finally generate a first gray image as input data for the machine vision application. In this way, the machine vision application is executed by using the first gray image with low resolution and no color, which can reduce the algorithm power consumption of the head-mounted device 10 in data processing. In this embodiment, the pixel merging mode can be 2×2, and the resolution of the first gray image after pixel merging and graying can be 800×650.
[0057] Optionally, the order of step S151 and step S152 may also be interchanged. That is, after acquiring the color image, the color image may be grayscaled first, and then the grayscaled color image may be pixel-merged to generate a first grayscale image for use as input data for the processor 300 to execute the machine vision application.
[0058] Please combine Figures 9 and 10 See also Figure 11 to Figure 12 , Fig.11 This is an example of pixel merging for a color image. Fig.12 yes Fig.11 Another example of pixel merging for a color image in .
[0059] Pixel merging for color images can be divided into two methods: software merging and hardware merging. Among them, software merging means that the image sensor 201 outputs the original image ( Fig.11 After the original image is processed by image signal processing (ISP), the original image processed by ISP is downsampled by 2×2 through a software algorithm to accumulate the pixel values of adjacent pixels in the original image, thereby obtaining a color image with reduced resolution ( Fig.11 Hardware merging means that after the adjacent pixels of the image sensor 201 are exposed, the charge accumulation process is completed in the image sensor 201 as 2×2 accumulation, and then the image sensor 201 directly outputs the original image with reduced resolution ( Fig.12 After ISP processing, a color image with reduced resolution is obtained ( Fig.12(shown in d).
[0060] Specifically, when the color image is pixel-merged by software merging, step S151 may include:
[0061] S1511, performing pixel merging on a color image that executes an image display application.
[0062] Since the software merging is achieved by downsampling through a software algorithm after the camera 200 outputs a color image and performs ISP processing, when the color image is used as input data for an image display application, pixel merging can also be performed at the same time, and a first grayscale image can be generated after subsequent grayscale processing. That is, the color image for executing the image display application and performing pixel merging can be the same frame. For example, the camera 200 can output 60 frames of color images, and each frame of the color image can be used as input data for the processor 300 to execute the image display application to meet the picture smoothness required for color perspective imaging. At the same time, each frame of the color image output by the camera 200 can be pixel merged to generate a first grayscale image as input data for the processor 300 to execute the machine vision application after subsequent grayscale processing. With such a setting, the picture smoothness of the color perspective imaging of the head-mounted device 10 (at least 60 frames) can be ensured, and the requirements of the head-mounted device 10 for executing machine vision applications can be met.
[0063] Optionally, considering that the frame rate requirement of the machine vision application is lower than that of the image display application, some frames in the color image for executing the image display application may not be pixel-merged to reduce the power consumption of the algorithm of the head-mounted device 10 in data processing. For example, the camera 200 can output 60 frames of color images, and each frame of the color image can be used as input data for the processor 300 to execute the image display application. At the same time, among the 60 frames of color images output by the camera 200, the color images of the even frames or odd frames can be pixel-merged and gray-scaled to generate 30 frames of first grayscale images as input data for the processor 300 to execute the machine vision application, and the output frame rate of 30 frames can also meet the requirements of the machine vision application.
[0064] Optionally, in addition to pixel merging of color images of even frames or odd frames, some frames may be selected from the multiple frames of color images output by the camera 200 for pixel merging in other random and fixed manners. It is only necessary that the frame rate of the first grayscale image finally output can meet the requirements of the machine vision application. This embodiment is not limited to this.
[0065] When the color image is pixel-merged by hardware merging, step S151 may include:
[0066] S1512, executing an image display application based on a portion of the color images in the plurality of frames of color images.
[0067] S1513, performing pixel combination on another part of the color images in the multiple frames of color images.
[0068] Since the hardware merging is directly implemented and outputted inside the image sensor 201 of the camera 200, the image sensor 201 can only output one of the color image used to execute the image display application and the color image for pixel merging in the same frame. That is, the color image for executing the image display application and the color image for pixel merging are different frames. For example, the camera 200 can output 120 frames of color images, and 60 frames of color images (equivalent to part of the color images in the multi-frame color image) can be used to execute the image display application, while the remaining 30 frames (equivalent to another part of the color images in the multi-frame color image) of the color image can be a low-resolution color image outputted after completing the pixel merging inside the camera 200, so as to be used as input data for the processor 300 to execute the machine vision application after grayscale processing.
[0069] Optionally, the color image for executing the image display application and the color image for pixel merging can be adjacent frames. That is, the color images belonging to the even-numbered frames among the multiple color image frames output by the camera 200 can be used to execute the image display application, while the color images belonging to the odd-numbered frames are low-resolution color images that have completed pixel merging inside the image sensor 201, so as to execute the machine vision application as input data. Alternatively, the color images belonging to the odd-numbered frames among the multiple color image frames output by the camera 200 can be used to execute the image display application, while the color images belonging to the even-numbered frames are low-resolution color images that have completed pixel merging inside the image sensor 201, so as to execute the machine vision application as input data.
[0070] Optionally, in addition to adjacent frames, some frames can be selected from the multiple frames of color images output by the camera 200 in other random or fixed manners to execute image display applications, while other frames are subjected to pixel merging. It is only necessary that the color image for executing the image display application and the color image for performing pixel merging are different frames, and this embodiment is not limited to this.
[0071] Optionally, considering that the camera 200 uses hardware merging when the output frame rate is 60 frames, there may be a problem of low frame rate affecting the smoothness of the picture, and frame interpolation processing can be performed subsequently through software algorithms to meet the frame rate required for executing image display applications.
[0072] Further, after the first grayscale image is generated, the first grayscale image can be used as input data for the processor 300 to execute the machine vision application. Fig. 9 and Fig.10As shown, the processor 300 can perform machine vision applications such as gesture recognition, 3D measurement, and 6DoF tracking based on the first grayscale image.
[0073] Please combine 6 to Figure 7 See also Figure 13 to Figure 14 , Fig.13 is another flowchart of the control method of the head mounted device 10 provided in the present application, Fig.14 1 is another flowchart of data transmission of the head mounted device 10 provided by the present application. In this embodiment, the execution subject of the control method of the head mounted device 10 is also the head mounted device 10 in the above embodiment.
[0074] like Fig.13 As shown, the control method provided in this embodiment includes the following steps:
[0075] S11, acquiring a second grayscale image.
[0076] S12, executing a machine vision application based on the second grayscale image.
[0077] S13, obtaining a color image.
[0078] S14, executing an image display application based on the color image.
[0079] S15, performing pixel merging and grayscale processing on the color image in a preset order to generate a first grayscale image.
[0080] S16, executing a machine vision application based on the first grayscale image.
[0081] The difference from the above-mentioned embodiment is that the control method provided in this embodiment further includes step S11 and step S12. Only step S11 and step S12 are described in detail below.
[0082] The second grayscale image can be formed by the camera 200 receiving the infrared light imaging emitted by the laser transmitter 400 into the real environment, and the second grayscale image can match the binocular structured light algorithm to increase the 3D structured light data for the processor 300 to perform 3D measurement machine vision applications. Compared with the first grayscale image in the aforementioned embodiment, the accuracy of performing 3D measurement using the second grayscale image is higher, and it can effectively measure the 3D information of the scene within the field of view, which is helpful for rendering the correct occlusion relationship between virtual and real objects in the scene during color perspective imaging. Of course, the second grayscale image can also be used as input data for the processor 300 to perform other machine vision applications, such as gesture recognition, 6DoF tracking, etc., which is not limited in this embodiment.
[0083] Specifically, step S11 may include:
[0084] S111, capturing an image having a color portion and an infrared portion.
[0085] S112, generating a second grayscale image based on the infrared part.
[0086] In this embodiment, the image sensor 201 may have a filter of an RGB-IR array, so that the image sensor 201 can simultaneously receive primary color light and infrared light in the real environment to capture a color portion A and an infrared portion B ( Figure 6 In order to obtain the second grayscale image, the infrared part B can be sampled separately during the ISP2 processing and combined to form the second grayscale image (as shown in FIG. Fig.14 At the same time, considering that the machine vision application does not have a high requirement for resolution, the infrared part B can directly output a second grayscale image with a resolution of 800×650 after being processed by ISP2, so as to be used as input data for the processor 300 to execute the machine vision application. Through the above settings, the image sensor 201 of the RGB-IR array can be used to obtain the second grayscale image.
[0087] Optionally, in addition to obtaining the second grayscale image through the above steps S111 and S112, the second grayscale image can also be obtained through other steps. For example, step S11 can also include the following steps:
[0088] S113, capturing a first image having a color portion.
[0089] S114, capturing a second image having a color portion and an infrared portion.
[0090] S115, determining a pixel value of the infrared portion based on a pixel difference between the first image and the second image to generate a second grayscale image.
[0091] In this embodiment, the image sensor 201 may have a filter of an RGB-W array, so that the image sensor 201 can simultaneously receive primary color light and infrared light in the real environment. Among them, since the W channel in the RGB-W array can pass primary color light and infrared light at the same time, the pixel values of the color part A and the infrared part B in the W channel will be superimposed, and the infrared part B cannot be directly used alone to form a second grayscale image. Based on this, in this embodiment, in the same scene, the first image with the color part A (the laser emitter 400 is not turned on, and there is no infrared light in the real environment), and the second image with the color part A and the infrared part B (the laser emitter 400 is turned on, and there is infrared light in the real environment) can be obtained respectively, and then the pixel value of the infrared part B is determined based on the pixel difference between the first image and the second image in the W channel, so as to sample the infrared part B separately and combine to generate the second grayscale image. Through the above settings, the image sensor 201 of the RGB-W array can also be used to obtain the second grayscale image.
[0092] Furthermore, step S13 may also include:
[0093] S131, generating a color image based on the color part.
[0094] Since the Bayer array of the image sensor 201 is changed to an RGB-IR array, the color part A captured by the image sensor 201 after exposure is missing (the primary color light information of the corresponding IR channel is missing). In order to ensure that the color image generated based on the color part A has a high resolution, the separated color part A needs to be interpolated and filled in ISP1 to fill the missing primary color light information of the color part A. Similarly, when the image sensor 201 is an RGB-W array, when a color image is generated based on the color part A, the separated color part A can also be interpolated and filled to generate a high-resolution color image.
[0095] The control method of the head mounted device 10 provided in the present application performs pixel merging and grayscale processing on the acquired color image in a preset order, so that the color image can be converted into a first grayscale image with a resolution lower than the color image, so as to execute a machine vision application based on the first grayscale image. In this way, the head mounted device 10 is only equipped with a color camera 200 capable of acquiring color images, and can also obtain grayscale images for executing machine vision applications, which is conducive to reducing the number of cameras 200 and reducing the cost, weight and power consumption of the head mounted device 10.
[0096] See also Fig.15 , Fig.15 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application.
[0097] like Fig.15 As shown, the computer-readable storage medium 900 stores program data 910, and when the program data 910 is executed by the processor, it is used to implement the steps in any of the above method embodiments. Among them, the computer-readable storage medium 900 can be a medium that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, or it can also be a server storing the computer program, and the server can send the stored computer program to other devices for execution, or it can also run the stored computer program by itself.
[0098] The above descriptions are only some embodiments of the present application, and do not limit the protection scope of the present application. Any equivalent device or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for controlling a head mounted device, characterized in that: The control method comprises: Get color images; Performing pixel merging and grayscale processing on the color image in a preset order to generate a first grayscale image; A machine vision application is performed based on the first grayscale image.
2. The control method according to claim 1, characterized in that: The steps of performing pixel merging and grayscale processing on the color image in a preset order include: performing pixel merging on the color image; Grayscale processing is performed on the color image after pixel merging.
3. The control method according to claim 2, characterized in that: The step of obtaining a color image also includes: An image display application is executed based on the color image.
4. The control method according to claim 3, characterized in that: The step of performing pixel merging on the color image comprises: Pixel merging is performed on the color image for executing the image display application.
5. The control method according to claim 3, characterized in that: The step of performing pixel merging on the color image comprises: executing an image display application based on a portion of the color images in the plurality of frames of the color images; Pixel combination is performed on another part of the color images in the multiple frames of color images.
6. The control method according to claim 1, characterized in that: Before the step of acquiring the color image, the method further includes: Acquire a second grayscale image; The machine vision application is executed based on the second grayscale image.
7. The control method according to claim 6, characterized in that: The step of acquiring the second grayscale image comprises: capturing an image having a color portion and an infrared portion; The second grayscale image is generated based on the infrared portion.
8. The control method according to claim 6, characterized in that: The step of acquiring the second grayscale image comprises: capturing a first image having a color portion; capturing a second image having the color portion and the infrared portion; The pixel value of the infrared portion is determined based on the pixel difference between the first image and the second image to generate the second grayscale image.
9. The control method according to claim 7 or 8, characterized in that: The steps to obtain a color image include: The color image is generated based on the color portion.
10. The control method according to claim 6, characterized in that: The pixel merging mode is 2×2, and the resolution of the color image is 1600×1300, and the resolution of the first grayscale image and the second grayscale image is 800×650.
11. A head mounted device, characterized in that: The head mounted device comprises: a head display mechanism, and a camera and a processor arranged on the head display mechanism; The camera is configured to capture a color image, and the head display mechanism is configured to display the color image; the processor is configured to perform pixel merging and grayscale processing on the color image in a preset order to generate a first grayscale image, and execute a machine vision application based on the first grayscale image.
12. The head mounted device according to claim 11, wherein: The number of the cameras is two, which are respectively a first camera and a second camera; the first camera and the second camera are arranged at intervals on the mounting side of the head display mechanism, and the mounting side has a top and a bottom that are arranged opposite to each other, and side portions respectively connected to the top and the bottom; wherein, The optical axis of at least one of the first camera and the second camera is tilted toward the direction close to the side portion to expand the coverage of the field of view angle of the first camera and the second camera.
13. The head mounted device according to claim 12, wherein: The optical axis of at least one of the first camera and the second camera is also tilted toward a direction close to the bottom.
14. The head mounted device according to claim 11, wherein: The head mounted device further comprises: a laser transmitter mounted on the head mounted display mechanism; wherein, The laser emitter is configured to emit infrared light; the camera is also configured to simultaneously capture an image having a color portion and an infrared portion to obtain the color image and a second grayscale image, respectively; the processor is also configured to execute the machine vision application based on the second grayscale image.
15. The head mounted device according to claim 11, wherein: The machine vision application includes at least one of 6DoF tracking, gesture recognition and 3D measurement.
16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and when the program data is executed by a processor, it is used to implement the control method according to any one of claims 1 to 10.
