Photographing method, device and equipment of head-mounted device and medium

Abstract

This disclosure provides a method, apparatus, device, and medium for taking pictures using a head-mounted device, applied to a control device. The method includes: receiving a first image output from an image sensor of the head-mounted device in response to a photo-taking command; and performing image processing on the first image using a programmable processor of the control device to generate a second image corresponding to the first image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a method for taking pictures with a head-mounted device, a device for taking pictures with a head-mounted device, a control device, and a computer-readable storage medium. Background Technology

[0002] Power consumption is a key indicator for head-mounted devices, and the camera is the main high-power component of head-mounted devices. The instantaneous current can reach 1A when taking a picture, which is the core cause of the sudden increase in the power consumption of the whole device. Optimizing the power consumption of the camera is the key to reducing the power consumption of AR products.

[0003] In related technologies, the photo-taking function of head-mounted devices performs the entire image processing process through their own hardware image processing module. This not only increases the hardware cost and weight of the device, but also significantly aggravates power consumption, which contradicts the design requirements of lightweight and low power consumption of head-mounted devices. If power consumption is reduced by simplifying the processing process, the image quality will be directly sacrificed, making it difficult to balance power consumption control and imaging effect. Summary of the Invention

[0004] The purpose of this disclosure is to provide a method, apparatus, device, and medium for taking pictures with a head-mounted device.

[0005] According to a first aspect of the present disclosure, a method for taking pictures using a head-mounted device is provided, applied to a control device, the method comprising: Receive the first image output from the image sensor of the head-mounted device in response to the photo capture command; The programmable processor of the control device performs image processing on the first image to generate a second image corresponding to the first image. Optionally, the method further includes: Based on the second image, generate image acquisition control parameters for taking the picture; The image acquisition control parameters are sent to the head-mounted device, which then acquires the next frame of image based on the image acquisition control parameters.

[0006] Optionally, generating image acquisition control parameters for taking a picture based on the second image includes: Extract statistical information from the second image; Based on the statistical information, a set control algorithm is executed to obtain the execution result; wherein, the set control algorithm includes at least one control algorithm among automatic exposure, automatic white balance and automatic focus; Based on the execution results, image acquisition control parameters for taking pictures are generated.

[0007] Optionally, the second image includes multiple sub-images with different resolutions.

[0008] Optionally, the step of performing image processing on the first image through the programmable processor of the control device to generate a second image corresponding to the first image includes: The first image is subjected to RAW domain correction processing to obtain the corrected first image; The corrected first image is subjected to RAW domain enhancement processing to obtain the third image; RAW domain optimization processing is performed on the third image to obtain a fourth image corresponding to the third image; The fourth image is de-mosaiced to obtain a fifth image corresponding to the fourth image; The fifth image is subjected to RGB domain color processing to obtain the second image.

[0009] Optionally, the RAW domain correction processing includes at least one of bad pixel correction, lens shading correction, and white balance adjustment. The RAW domain enhancement processing includes at least one of green channel equalization correction and Bell domain noise reduction. The RAW domain optimization process includes at least one of tone mapping and color difference correction.

[0010] Optionally, the programmable processor includes at least one of a central processing unit (CPU), a programmable processor (GPU), a neural processing unit (NPU), and a digital signal processor (DSP). According to a third aspect of the present disclosure, a camera device for a head-mounted device is provided, applied to a control device, the device comprising: The receiving module is used to receive the first image output from the image sensor of the head-mounted device in response to the photo capture command; The processing module is used to perform image signal processing on the first image through the programmable processor of the control device to generate a second image corresponding to the first image.

[0011] According to a third aspect of the present disclosure, a control device is provided, the device comprising: Memory is used to store executable computer instructions; A processor, configured to execute the method described in accordance with the first aspect above, under the control of the executable computer instructions.

[0012] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, the computer program implementing the method according to the first aspect above when executed by a processor.

[0013] One beneficial effect of this disclosure is that, upon receiving a first image output by the image sensor of the head-mounted device in response to a photographing command, the control device processes the first image using its own programmable processor based on the photographing command to generate a second image corresponding to the first image. In other words, it improves the high-power image processing flow of the head-mounted device by sending the first image output by the head-mounted device's image sensor to the control device for image processing to generate the second image. This effectively reduces the power consumption of the head-mounted device. Furthermore, the control device performs image processing using its own programmable processor, exhibiting excellent computing power adaptability and effectively controlling its own power consumption without incurring additional power burden, thus balancing overall power control and image quality.

[0014] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0016] Figure 1 This is a schematic diagram of the hardware configuration of the camera system of a head-mounted device according to an embodiment of the present disclosure; Figure 2 This is a schematic flowchart of a head-mounted device for taking pictures according to an embodiment of the present disclosure; Figure 3 This is a schematic flowchart of a photographing method using a head-mounted device according to another embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating the interaction process between a head-mounted device and a control device, based on an example. Figure 5 This is a schematic block diagram of a camera device for a head-mounted device according to an embodiment of the present disclosure; Figure 6 This is a schematic block diagram of a control device according to an embodiment of the present disclosure. Detailed Implementation

[0017] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.

[0018] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0019] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0020] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0022] <Hardware Configuration> Figure 1 This is a schematic diagram of the hardware configuration of a camera system for a head-mounted device according to an embodiment of the present disclosure. Figure 1 A head-mounted device 100 and a control device 200 are shown. The head-mounted device 100 and the control device 200 are communicatively connected.

[0023] In one embodiment of this disclosure, such as Figure 1 As shown, the head-mounted device 100 may include at least a processor 101, a memory 102, a communication device 103, and a camera device 104. Furthermore, the head-mounted device 100 may also include a power management chip 105 and a battery 106. It should be noted that the head-mounted device 100 may or may not have a display device (not shown in the figure), wherein the display device may be a screen.

[0024] The processor 101 can be a programmable processor. The memory 102 can store the underlying software, system software, application software, data, etc., required for the operation of the head-mounted device 100. The memory 102 can include various forms of memory, such as Double Data Rate Synchronous Dynamic Random Access Memory (DDR). The communication device 103 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, and 5G communication device. The camera device 104 can be a single camera or multiple cameras. The power management chip 105 is used to manage the power input to the head-mounted device 100 and can also manage the battery 106 to ensure high utilization efficiency. The battery 106 is a lithium-ion battery, etc.

[0025] The head-mounted device 100 may be a helmet or glasses of the type of extended reality (XR), such as augmented reality (AR), virtual reality (VR), or mixed reality (MR), and this disclosure does not limit this type. Figure 1 The components shown are merely illustrative. The head-mounted device 100 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The head-mounted device 100 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0026] In one embodiment of this disclosure, such as Figure 1 As shown, the control device 200 may include a processor 201, a memory 202, a communication device 203, a display device 204, a camera device 205, etc. Furthermore, the control device 200 may also include a power management chip 206 and a battery 207, etc.

[0027] The processor 201 can be a programmable processor. The memory 202 can store the underlying software, system software, application software, data, etc., required for the operation of the control device 200. The memory 202 can include various forms of memory, such as DDR. The communication device 203 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, or 5G communication device. The display device 204 can be a touchscreen. The camera device 205 can be a single camera or multiple cameras. The power management chip 206 is used to manage the power supply of the input control device 200 and can also manage the battery 207 to ensure maximum utilization efficiency. The battery 207 is a lithium-ion battery, etc.

[0028] The control device 200 can be an Android device, such as a mobile phone, tablet computer, PDA, or PUCK adapted to the head-mounted device 100. Figure 1 The components shown are merely illustrative. The control device 200 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The control device 200 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0029] In this embodiment, the memory 202 of the control device 200 is used to store computer instructions, and the processor 201 is used to retrieve the computer instructions from the memory 202 to execute the imaging method of the head-mounted device applied to the control device. Those skilled in the art can design the instructions according to the scheme disclosed in this invention. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0030] It should be understood that, despite Figure 1 Only one head-mounted device 100 and one control device 200 are shown, but this does not mean that the number of each is limited. The head-mounted device's photography system may contain multiple head-mounted devices 100 and multiple control devices 200.

[0031] In the above description, those skilled in the art can design instructions based on the solutions provided in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0032] <Method Implementation> Figure 2 This is a schematic flowchart of a head-mounted device for taking pictures according to an embodiment of the present disclosure. The head-mounted device's method for taking pictures can be... Figure 1 The control equipment shown is implemented. For example... Figure 2 As shown, the photographing method of the head-mounted device in this embodiment may include the following steps S2100 to S2200: Step S2100: Receive the first image output from the image sensor of the head-mounted device in response to the photo capture command.

[0033] The photo-taking command can be a control command used to trigger the head-mounted device to perform image acquisition. The triggering method of the photo-taking command can be flexibly set according to the actual usage scenario of the head-mounted device, including but not limited to triggering by local physical buttons on the head-mounted device, touch interaction triggering, voice command triggering, and remote triggering commands sent by the control device to the head-mounted device.

[0034] The first image is usually the raw image directly captured and output by the image sensor of the head-mounted device after receiving the shooting command. It can also be called a RAW image, such as a Bayer format RAW image.

[0035] In this embodiment, when the head-mounted device is working normally and maintaining a communication connection with the control device, after receiving a photo-taking command, the head-mounted device will control its image sensor to output a first image based on the current image acquisition control parameters and temporarily store it in the local cache module. It should be noted that the head-mounted device can be pre-configured for both preview and photo-taking scenarios; the preview is used for low-cost local display, and the photo-taking is used to acquire the first image.

[0036] While generating and temporarily storing the first image, the head-mounted device verifies the validity of the communication link with the control device. The two establish a dedicated data transmission link based on preset wireless communication protocols such as Bluetooth. This link can be a bidirectional link to ensure the stability and low latency of the transmission.

[0037] The head-mounted device sends a first image through a wireless communication transmitting module. The control device receives the first image transmitted by the head-mounted device through the aforementioned link through a wireless communication receiving module, and simultaneously performs integrity and validity verification on the first image. After completing the verification, the control device temporarily stores the first image in its own cache unit.

[0038] In this embodiment, the head-mounted device can synchronously transmit the first image and its corresponding metadata to the control device. The metadata may include image acquisition control parameters, lens and hardware parameters, and scene-related information.

[0039] After performing the above step S2100, receiving the first image output from the image sensor of the head-mounted device in response to the image capture command, proceeds to: Step S2200: The programmable processor of the control device performs image processing on the first image to generate a second image corresponding to the first image.

[0040] A programmable processor is a processing unit that has the ability to execute instructions and can implement specific data processing and arithmetic functions through software programs or firmware configuration. Its processing logic can be dynamically adjusted through program instructions, rather than being fixedly implemented by hardware circuits. A programmable processor can include at least one of a central processing unit (CPU), a programmable GPU, a neural network processor (NPU), and a digital signal processor (DSP).

[0041] Programmable processors may also include various general-purpose or special-purpose processing units, such as Field Programmable Gate Arrays (FPGAs), Tensor Processing Units (TPUs), and Vision Processing Units (VPUs). These processing units can all complete the image processing flow described in this disclosure by executing software instructions or configuring hardware logic. The core feature of this type of programmable processor is the programmability of its processing functions, which clearly distinguishes it from Application Specific Integrated Circuits (ASICs) with fixed processing flows. Image Signal Processors (ISPs) are a typical example of this type of ASIC.

[0042] The second image can be an image generated by the control device after performing image processing on the first image output by the head-mounted device through a programmable processor. This second image is the image captured in this photo-taking process.

[0043] In this embodiment, the control device can call its own programmable processor to perform image processing on the received and verified first image, and finally generate a second image corresponding to the first image.

[0044] According to embodiments of this disclosure, when the control device receives a first image output by the image sensor of the head-mounted device in response to a photo-taking command, it performs image processing on the first image using its own programmable processor based on the photo-taking command to generate a second image corresponding to the first image. In other words, it improves the high-power image processing flow of the head-mounted device by sending the first image output by the head-mounted device's image sensor to the control device for image processing to generate the second image. This effectively reduces the power consumption of the head-mounted device. Furthermore, the control device performs image processing using its own programmable processor, exhibiting excellent computing power adaptability and effectively controlling its own power consumption without incurring additional power burden, thus balancing overall power control and image quality.

[0045] In one embodiment, the above step S2200, which processes the first image using the programmable processor of the control device to generate a second image corresponding to the first image, may further include the following steps S2210 to S2250: Step S2210: Perform RAW domain correction processing on the first image to obtain the corrected first image.

[0046] The RAW domain correction processing can include at least one of the following: bad pixel correction, lens shading correction, and white balance adjustment, used to correct the distortion generated in the first image (such as a Bayer format RAW image) during hardware acquisition, so as to ensure the image quality of the final generated second image.

[0047] Specifically, the above-mentioned defect correction can be performed as follows: detect abnormal pixels in the first image, including but not limited to overly bright pixels (pure white pixels), overly dark pixels (pure black pixels), and flickering pixels; for each detected abnormal pixel, perform interpolation operation using its neighboring normal pixels, and replace the original data of the abnormal pixel with the interpolation result to ensure the continuity and integrity of image pixels.

[0048] Due to the limitations of lens optical characteristics, image sensors are prone to vignetting when acquiring images. This phenomenon occurs when the brightness of the central area of ​​the image is higher than that of the edges and corners, resulting in uneven brightness distribution. Lens vignetting correction can be achieved by: first, using pre-calibrated vignetting correction coefficients, which correspond one-to-one with the pixel positions in the image; then, matching the corresponding correction coefficients to the coordinates of each pixel in the first image (central area, edge area, corner area), dynamically adjusting the brightness gain of that pixel, and compensating for the brightness of the dark areas at the edges and corners. This makes the overall brightness distribution of the first image more uniform, eliminating image quality distortion caused by vignetting.

[0049] Due to the varying color temperatures of the light sources in the shooting scene, the first image is prone to color cast, resulting in inaccurate color reproduction. The white balance adjustment described above can be achieved by: determining the color temperature deviation of the current scene based on the metadata corresponding to the first image; and adjusting the brightness gain of the R (red), G (green), and B (blue) channels in the first image according to the color temperature deviation to correct the imbalance of each color channel, ensuring that the white areas in the image present a standard neutral color, and ensuring that the image color reproduction conforms to human visual habits and closely resembles the colors of the real scene.

[0050] Through the RAW domain correction processing in step S2210, the control device can effectively correct the pixel distortion, brightness distortion and color distortion generated in the first image at the hardware acquisition level, and output the first image after distortion correction, which can effectively ensure the image quality accuracy and visual effect of the final second image.

[0051] Step S2220: Perform RAW domain enhancement processing on the corrected first image to obtain the third image.

[0052] The RAW domain enhancement processing can include at least one of green channel equalization correction and Bayer domain noise reduction, which can further optimize image quality within Bayer format RAW images, suppress noise, correct channel brightness unevenness, avoid amplification distortion in subsequent processing, and improve the quality of the final second image.

[0053] Because the green channel has a high pixel proportion in Bayer format RAW images, it directly determines the image brightness and detail, and is prone to uneven brightness distribution due to sensor characteristics and lighting conditions. The aforementioned green channel equalization correction can specifically involve: statistically analyzing the brightness value of the green channel and using an adaptive equalization algorithm to adjust this brightness value, making the brightness of the green channel more uniform.

[0054] Since the first image after RAW domain correction still retains sensor noise, directly entering color reconstruction would amplify graininess and false colors. The aforementioned Bayer domain denoising can be specifically implemented as follows: denoising is performed directly on the Bayer format RAW image; noise and image details are distinguished based on adjacent pixels in the same channel; an adaptive algorithm is used to smooth the noise, improving image purity while preserving edge details, resulting in higher computational efficiency.

[0055] The third image obtained through this step S2220 is still a Bayer format RAW image. Compared with the first image after RAW domain correction, it has more uniform brightness and lower noise.

[0056] Step S2230: Perform RAW domain optimization processing on the third image to obtain a fourth image corresponding to the third image.

[0057] The RAW domain optimization processing can include at least one of tone mapping and color difference correction. The color difference correction can be lateral color difference correction, which can further optimize the dynamic range of the image and correct color distortion in the Bayer domain, thereby further improving the imaging effect.

[0058] Specifically, the tone mapping described above can be: generating a mapping curve based on the histogram or local statistical results of the third image, and applying the mapping curve to the third image to compress the image highlights and enhance the details in the shadows, thereby adapting to subsequent display or encoding requirements.

[0059] Specifically, the above-mentioned color difference correction can be performed as follows: based on lens calibration data, the red and blue channels are geometrically aligned relative to the green channel to eliminate color vignetting at the image edges and improve image sharpness.

[0060] The fourth image obtained through this step S2230 is still a Bayer format RAW image, with a more reasonable dynamic range and cleaner color edges, providing a better data foundation for subsequent color reconstruction.

[0061] Step S2240: Perform demosaic processing on the fourth image to obtain the fifth image corresponding to the fourth image.

[0062] The fifth image can be an RGB image.

[0063] Specifically, the above demosaicing process can be performed on the RAW image in the Bayer domain. For example, an adaptive algorithm can be used to preserve image details in the edge areas and balance computational efficiency in the flat areas of the image, ultimately reconstructing a full-resolution RGB image.

[0064] In step S2240, the control device performs demosaicing and RGB reconstruction processing on the Bayer format RAW image and outputs an RGB image.

[0065] Step S2250: Perform RGB domain color processing on the fifth image to obtain the second image.

[0066] The second image can be a YUV image, where Y represents the luminance component and U and V represent the chrominance components. This second image can be a full-resolution YUV image.

[0067] In step S2250, the control device can perform color space conversion, brightness adjustment, contrast adjustment, and saturation enhancement on the RGB image. It can also further perform global or local tone mapping in the RGB domain to optimize highlight and shadow details and improve image depth. After processing, the image is converted to YUV 4:2:2 or 4:2:0 format for subsequent image encoding, storage, or display output.

[0068] Through the embodiments disclosed herein, the above image processing flow can be executed by the programmable processor of the control device, and the final output YUV format image can be directly adapted for image encoding, storage and high-definition display. Overall, multiple optimizations are achieved in image quality, power consumption, processing efficiency and scene adaptability, which greatly improves the practicality and user experience of head-mounted device photography solutions.

[0069] In one embodiment, the second image may include multiple sub-images with different resolutions.

[0070] The multiple sub-images with different resolutions can include full-resolution images, medium-resolution images, and low-resolution images. For example, a full-resolution YUV image, a medium-resolution YUV image, and a low-resolution YUV image together constitute a second image to adapt to different scenario requirements such as high-definition storage, real-time preview, fast transmission, and intelligent analysis.

[0071] Figure 3 This is a schematic flowchart of a photographing method using a head-mounted device according to another embodiment of this disclosure. Figure 3 As shown, in this embodiment, after performing the above step S2200 to process the first image through the programmable processor of the control device and generate a second image corresponding to the first image, the head-mounted device imaging method of this embodiment further includes the following steps S3100 to S3200: Step S3100: Based on the second image, generate image acquisition control parameters for taking a picture.

[0072] Specifically, the image acquisition control parameters for taking pictures can be generated based on the second image in the following way: extract statistical information from the second image; execute a set control algorithm based on the statistical information to obtain the execution result; and generate the image acquisition control parameters for taking pictures based on the execution result.

[0073] The control algorithm can include at least one of the following: Auto Exposure (AE), Auto White Balance (AWB), and Auto Focus (AF).

[0074] The statistical information may include at least one of the following: histogram, color square, color temperature estimate, and image sharpness evaluation value.

[0075] The image acquisition control parameters used for taking pictures may include: exposure parameters, gain parameters, and focus parameters.

[0076] In one example, the control device can extract only the statistical information of the second image, and based on the statistical information, execute a set control algorithm to obtain the execution result, and generate image acquisition control parameters for taking pictures based on the execution result.

[0077] In another example, the control device can also extract the first statistical information of the first image and the second statistical information of the second image respectively, and based on the first statistical information and the second statistical information, execute a set control algorithm to obtain the execution result, and generate image acquisition control parameters for taking pictures according to the execution result.

[0078] Step S3200: The image acquisition control parameters are sent to the head-mounted device, and the head-mounted device acquires the next frame of image based on the image acquisition control parameters.

[0079] In this embodiment, the head-mounted device receives image acquisition control parameters returned by the control device. The image sensor can then directly use these returned image acquisition control parameters to perform imaging during subsequent image acquisition. For example, if the control device determines that the current scene brightness is insufficient based on the analysis of the previous frame image, it can generate and return image acquisition control parameters that increase the exposure time. After the head-mounted device configures these image acquisition control parameters to the image sensor, the brightness of subsequently acquired images can be precisely increased.

[0080] Through the embodiments disclosed herein, a closed loop of acquisition-processing-feedback-optimized acquisition can be formed, enabling the head-mounted device to adaptively optimize imaging quality frame by frame in dynamic environments such as changes in light and scene switching.

[0081] <Example> Next, refer to Figure 4 Taking AR glasses as the head-mounted device and PUCK as the control device as an example, an example of a head-mounted device taking pictures is shown. In this example, the head-mounted device taking pictures can further include: In step S401, after the AR glasses launch the camera application, the image sensor outputs a MIPI (Mobile Industry Processor Interface) data stream.

[0082] In step S402, the AR glasses receive the MIPI data stream through the Mobile Industry Processor Interface (CSID) and send it to the Image Front End (IFE) module. The IFE module processes the MIPI data stream and stores it in local DDR. The Image Process Engine (IPE) module retrieves the processed MIPI data stream from the local DDR, processes it, and stores it in local DDR. The image encoding module retrieves the processed MIPI data stream from the local DDR, processes it, and generates a preview image.

[0083] Step S403: The AR glasses receive the photo-taking command.

[0084] In step S404, the AR glasses respond to the photo-taking command by outputting a Bayer-format RAW image through the image sensor and storing the RAW image and metadata in the local DDR.

[0085] In step S405, the AR glasses obtain the Bayer-formatted RAW image and metadata from the local DDR, and send the Bayer-formatted RAW image and metadata to the PUCK via the WiFi connection between the AR glasses and the PUCK.

[0086] In step S501, PUCK receives the Bayer format RAW image sent by the AR glasses and stores it in the local DDR.

[0087] In step S501, the Puck device pre-starts the WiFi receiving service and is configured with the same image size parameters as the AR glasses device to ensure processing compatibility. The Puck device builds a software ISP processing pipeline through the GPU, which does not require connection to a physical camera, and the feasibility of the process can be verified in advance through test data.

[0088] In step S502, PUCK obtains the RAW image from the local DDR and performs bad pixel correction, lens shading correction, and white balance adjustment processing via GPU.

[0089] In step S503, PUCK further performs green channel equalization correction and Bell domain noise reduction processing via GPU.

[0090] In step S504, PUCK further performs tone mapping and color difference correction processing through the GPU, and outputs a full-resolution RGB image.

[0091] In step S505, PUCK performs RGB domain color processing and format conversion on the RGB image using the GPU to obtain the captured image.

[0092] The second image may include a full-resolution YUV image, a medium-resolution YUV image, and a low-resolution YUV image.

[0093] In step S506, PUCK's GPU generates image acquisition control parameters for taking pictures based on the captured image, and sends the image acquisition control parameters to the AR glasses.

[0094] Step S406: The AR glasses acquire the next frame of image based on the image acquisition control parameters.

[0095] In this example, by sending the RAW image of the AR glasses to PUCK's GPU for processing and image encoding, the power consumption of the AR glasses is effectively reduced.

[0096] <Device Embodiment> Figure 5 This is a schematic block diagram of a camera device for a head-mounted device according to an embodiment of the present disclosure, applied to a control device. (Refer to...) Figure 5 As shown, the head-mounted device's camera 500 includes a receiving module 510 and a processing module 520.

[0097] The receiving module 510 is used to receive the first image output from the image sensor of the head-mounted device in response to the photo capture command; The processing module 520 is used to perform image signal processing on the first image through the programmable processor of the control device to generate a second image corresponding to the first image.

[0098] In one embodiment, the device 500 includes a transmitting module (not shown).

[0099] The processing module 520 is also configured to generate image acquisition control parameters for taking a picture based on the second image; The sending module is used to send the image acquisition control parameters to the head-mounted device, and the head-mounted device performs the next frame image acquisition based on the image acquisition control parameters.

[0100] In one embodiment, the processing module 520 is specifically used to extract statistical information from the second image; based on the statistical information, execute a set control algorithm to obtain an execution result; wherein the set control algorithm includes at least one control algorithm among automatic exposure, automatic white balance and automatic focus; and generate image acquisition control parameters for taking pictures according to the execution result.

[0101] In one embodiment, the second image comprises multiple sub-images with different resolutions.

[0102] In one embodiment, the processing module 520 is specifically configured to perform RAW domain correction processing on the first image to obtain a corrected first image; perform RAW domain enhancement processing on the corrected first image to obtain a third image; perform RAW domain optimization processing on the third image to obtain a fourth image corresponding to the third image; perform de-mosaic processing on the fourth image to obtain a fifth image corresponding to the fourth image; and perform RGB domain color processing on the fifth image to obtain the second image.

[0103] In one embodiment, the RAW domain correction processing includes at least one of bad pixel correction, lens shading correction, and white balance adjustment. The RAW domain enhancement processing includes at least one of green channel equalization correction and Bell domain noise reduction. The RAW domain optimization process includes at least one of tone mapping and color difference correction.

[0104] In one embodiment, the programmable processor includes at least one of a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and a digital signal processor (DSP).

[0105] According to embodiments of this disclosure, when the control device receives a first image output by the image sensor of the head-mounted device in response to a photo-taking command, it performs image processing on the first image using its own programmable processor based on the photo-taking command to generate a second image corresponding to the first image. In other words, it improves the high-power image processing flow of the head-mounted device by sending the first image output by the head-mounted device's image sensor to the control device for image processing to generate the second image. This effectively reduces the power consumption of the head-mounted device. Furthermore, the control device performs image processing using its own programmable processor, exhibiting excellent computing power adaptability and effectively controlling its own power consumption without incurring additional power burden, thus balancing overall power control and image quality.

[0106] <Equipment Example> Figure 6 This is a schematic block diagram of a control device according to an embodiment of the present disclosure. Figure 6 As shown, the control device 600 includes a processor 610 and a memory 620.

[0107] The memory 620 can be used to store executable computer instructions.

[0108] The processor 610 can be used to execute the photographing method of the head-mounted device according to the method embodiments of this disclosure, under the control of the executable computer instructions.

[0109] The control device 600 can be as follows: Figure 1 The control device 200 shown.

[0110] In another embodiment, the control device 600 may include the camera device 500 of the head-mounted device described above.

[0111] In one embodiment, each module of the camera device 500 of the head-mounted device can be implemented by the processor 610 running computer instructions stored in the memory 620.

[0112] Computer-readable storage media This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the head-mounted device photography method provided in this disclosure.

[0113] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0114] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0115] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0116] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0117] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0119] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0121] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for taking photos using a head-mounted device, characterized in that, Applied to control equipment, the method includes: Receive the first image output from the image sensor of the head-mounted device in response to the photo capture command; The programmable processor of the control device performs image processing on the first image to generate a second image corresponding to the first image.

2. The method according to claim 1, characterized in that, The method further includes: Based on the second image, generate image acquisition control parameters for taking the picture; The image acquisition control parameters are sent to the head-mounted device, which then acquires the next frame of image based on the image acquisition control parameters.

3. The method according to claim 2, characterized in that, The step of generating image acquisition control parameters for taking a picture based on the second image includes: Extract statistical information from the second image; Based on the statistical information, a set control algorithm is executed to obtain the execution result; wherein, the set control algorithm includes at least one control algorithm among automatic exposure, automatic white balance and automatic focus; Based on the execution results, image acquisition control parameters for taking pictures are generated.

4. The method according to claim 1, characterized in that, The second image comprises multiple sub-images with different resolutions.

5. The method according to claim 1, characterized in that, The step of performing image processing on the first image through the programmable processor of the control device to generate a second image corresponding to the first image includes: The first image is subjected to RAW domain correction processing to obtain the corrected first image; The corrected first image is subjected to RAW domain enhancement processing to obtain the third image; RAW domain optimization processing is performed on the third image to obtain a fourth image corresponding to the third image; The fourth image is de-mosaiced to obtain a fifth image corresponding to the fourth image; The fifth image is subjected to RGB domain color processing to obtain the second image.

6. The method according to claim 5, characterized in that, The RAW domain correction processing includes at least one of the following: bad pixel correction, lens shading correction, and white balance adjustment. The RAW domain enhancement processing includes at least one of green channel equalization correction and Bell domain noise reduction. The RAW domain optimization process includes at least one of tone mapping and color difference correction.

7. The method according to any one of claims 1 to 6, characterized in that, The programmable processor includes at least one of a central processing unit (CPU), a programmable GPU, a neural network processor (NPU), and a digital signal processor (DSP).

8. A head-mounted camera device, characterized in that, Applied to control equipment, the device includes: The receiving module is used to receive the first image output from the image sensor of the head-mounted device in response to the photo capture command; The processing module is used to perform image signal processing on the first image through the programmable processor of the control device to generate a second image corresponding to the first image.

9. A control device, characterized in that, The control device includes: Memory is used to store executable computer instructions; A processor configured to execute the method according to any one of claims 1-7, under the control of the executable computer instructions.

10. A computer-readable storage medium, wherein, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1-7.

Images