Methods, equipment, systems and media for taking pictures with head-mounted devices

Abstract

This disclosure provides a method, device, system, and medium for taking pictures with a head-mounted device, relating to the field of head-mounted device technology. The method, when applied to a head-mounted device, includes: acquiring a first set of original images upon receiving a picture-taking command; determining the current environmental scene category based on the first set of original images, including high dynamic range (HDR) and low dynamic range (LVR) categories; determining the original image acquisition method and a first image preprocessing method corresponding to the current environmental scene category; processing a second set of original images acquired using the original image acquisition method according to the first image preprocessing method to obtain a first preprocessed original image; and sending the first preprocessed original image to a control device communicatively connected to the head-mounted device, whereby the control device, upon receiving the first preprocessed original image, generates a picture based on it. This method reduces the overall power consumption of the head-mounted device.

Description

Technical Field

[0001] This disclosure relates to the field of head-mounted display technology, and more specifically, to a method, apparatus, system, and medium for taking pictures with a head-mounted device. Background Technology

[0002] Power consumption directly affects the overall design, wearing comfort, and lightweight design of head-mounted devices. Therefore, power consumption has become a key indicator for evaluating head-mounted devices.

[0003] Display screens and cameras are key components affecting the power consumption of head-mounted devices. Currently, mainstream head-mounted devices have abandoned displays, retaining only the camera's photo-taking function. However, during the photo-taking process, the camera generates significant power consumption due to ISP processing and encoding, becoming the main source of power consumption for head-mounted devices. Therefore, reducing the power consumption of the photo-taking function can effectively reduce the overall power consumption of head-mounted devices, which has become an urgent technical problem to be solved. Summary of the Invention

[0004] One object of this disclosure is to provide a new technical solution for taking pictures with head-mounted devices.

[0005] According to a first aspect of this disclosure, a method for taking pictures using a head-mounted device is provided, applied to the head-mounted device, the method comprising: Upon receiving a photo-taking command, the first set of raw images is captured; Based on the first original image group, the current environment scene category is determined, and the current environment scene category includes a high dynamic range category and a low dynamic range category; Determine the original image acquisition method and the first image preprocessing method corresponding to the current environmental scene category; According to the first image preprocessing method, the second original image group acquired according to the original image acquisition method is processed to obtain the first preprocessed original image; The first preprocessed original image is sent to a control device that is communicatively connected to the head-mounted device. Upon receiving the first preprocessed original image, the control device generates a photographed image based on the first preprocessed original image.

[0006] Optionally, determining the original image acquisition method and the first image preprocessing method corresponding to the current environmental scene category includes: When the current environment scene category is high dynamic range category, the original image acquisition method is determined to be acquiring at least three frames of original images with different exposure values ​​and the first image preprocessing method is high dynamic range processing; When the current environment scene category is low dynamic range, the original image acquisition method is determined to be acquiring multiple frames of original images with the same exposure value, and the first image preprocessing method is multi-frame noise reduction processing.

[0007] Optionally, before processing the second set of original images acquired according to the original image acquisition method according to the first image preprocessing method to obtain the first preprocessed original image, the method further includes: Multiple frames of original images were acquired using the original image acquisition method to obtain the third set of original images; According to the second image preprocessing method, each frame of the original image in the third original image group is processed to obtain the second original image group. The second image preprocessing method is the image front-end processing method.

[0008] Optionally, determining the current environmental scene category based on the first original image group includes: Based on the first original image group, the current environmental scene category and the current image sharpness value are determined respectively; Sending the first preprocessed original image to a control device communicatively connected to the head-mounted device includes: The image compression method is determined based on the current image sharpness value; The first preprocessed original image is compressed into a compressed image according to the image compression method described above; The compressed image is sent to a control device that is communicatively connected to the head-mounted device.

[0009] Optionally, determining the image compression method based on the current image sharpness value includes: If the current image sharpness value is greater than the preset sharpness value, the image compression method is determined to be lossless compression. If the current image sharpness value is less than or equal to a preset sharpness value, the image compression method is determined to be visual lossless compression.

[0010] According to a second aspect of this disclosure, a method for taking pictures using a head-mounted device is provided, the method being applied to a control device communicatively connected to the head-mounted device, the method comprising: Receive the first preprocessed raw image sent by the head-mounted device; The Bayer processing stage and the image back-end processing are sequentially performed on the first preprocessed original image to obtain the photographed image.

[0011] Optionally, receiving the first preprocessed raw image sent by the head-mounted device includes: Receive a compressed image obtained by compressing the original image after the first preprocessing, sent by the head-mounted device; Decompress the compressed image to obtain the first preprocessed original image.

[0012] According to a third aspect of this disclosure, a photographing device for a head-mounted device is provided, wherein, when the photographing device for the head-mounted device is applied to a head-mounted device, the photographing device for the head-mounted device includes: The acquisition module is used to acquire the first group of raw images when a photo-taking command is received; The first determining module is used to determine the current environment scene category based on the first original image group, wherein the current environment scene category includes a high dynamic range category and a low dynamic range category; The second determining module is used to determine the original image acquisition method and the first image preprocessing method corresponding to the current environmental scene category; The first processing module is used to process the second original image group acquired according to the original image acquisition method according to the first image preprocessing method to obtain the first preprocessed original image. The sending module is used to send the first preprocessed original image to a control device that is communicatively connected to the head-mounted device. Upon receiving the first preprocessed original image, the control device generates a photographed image based on the first preprocessed original image. When the camera device of the head-mounted device is used in a control device, the camera device of the head-mounted device includes: The receiving module is used to receive the first preprocessed raw image sent by the head-mounted device; The second processing module is used to sequentially perform Bayer processing and image back-end processing on the first preprocessed original image to obtain the captured image.

[0013] According to a fourth aspect of this disclosure, an electronic device is provided, wherein, when the electronic device is a head-mounted device, the electronic device includes a first processor and a first memory, the first memory being configured to store computer instructions, and the first processor being configured to invoke the computer instructions from the first memory to perform the method as described in any one of the first aspects; When the electronic device is a control device, the electronic device includes a second processor and a second memory, the second memory being used to store computer instructions, and the second processor being used to retrieve the computer instructions from the second memory to perform the method as described in any one of the second aspects.

[0014] According to a fifth aspect of this disclosure, a head-mounted camera system is provided, comprising a head-mounted device as described in the fourth aspect and a control device as described in the fourth aspect.

[0015] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first or second aspects.

[0016] This disclosure provides a method for taking pictures using a head-mounted device. The method includes: acquiring a first set of original images upon receiving a picture-taking command; determining the current environmental scene category based on the first set of original images, including high dynamic range (HDR) and low dynamic range (LVR) categories; determining the original image acquisition method and a first image preprocessing method corresponding to the current environmental scene category; processing the second set of original images acquired using the first image preprocessing method according to the first image preprocessing method to obtain a first preprocessed original image; and sending the first preprocessed original image to a control device communicatively connected to the head-mounted device. Upon receiving the first preprocessed original image, the control device generates a picture based on the first preprocessed original image. In this method, the head-mounted device performs lightweight preprocessing to obtain an accurate first preprocessed original image, while the control device performs high-power, offline-processable picture encoding processing, effectively reducing the overall power consumption of the head-mounted device while maintaining image accuracy.

[0017] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0019] Figure 1 This is a schematic diagram of the hardware configuration of a head-mounted camera system provided in this disclosure; Figure 2 This is a flowchart illustrating a head-mounted device photography method disclosed herein. Figure 1 ; Figure 3 This is a flowchart illustrating a head-mounted device photography method disclosed herein. Figure 2 ; Figure 4 This is a flowchart illustrating a head-mounted device photography method disclosed herein. Figure 3 ; Figure 5 This is a schematic diagram of the structure of a head-mounted camera device provided in this disclosure. Figure 1 ; Figure 6 This is a schematic diagram of the structure of a head-mounted camera device provided in this disclosure. Figure 2 . Detailed Implementation

[0020] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0021] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0022] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0023] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0025] <Hardware Configuration> Figure 1 This is a schematic diagram of the hardware configuration of a head-mounted camera system provided in this disclosure. The head-mounted camera system 10 includes a head-mounted device 100 and a control device 200. The head-mounted device 100 and the control device 100 are communicatively connected.

[0026] In one embodiment of this disclosure, such as Figure 1 As shown, the head-mounted device 100 may include at least a first processor 101, a first memory 102, a first communication device 103, and a first camera device 104. Furthermore, the head-mounted device 100 may also include a first power management chip 105 and a first battery 106, etc. It should be noted that the head-mounted device 100 may or may not have a first display device (not shown in the figure), wherein the first display device may be a display screen.

[0027] The first processor 101 can be any of various processors, such as CPU, GPU, NPU, ISP, and DSP. The first memory 102 can store the underlying software, system software, application software, and data required for the operation of the head-mounted device 100. The first memory 102 can include various forms of memory, such as ROM, RAM, and Flash, where RAM can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR). The first communication device 103 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, or 5G communication device. The first camera device 104 can be a single camera or multiple cameras. The first power management chip 105 is used to manage the power input to the head-mounted device 100 and can also manage the first battery 106 to ensure high utilization efficiency. The first battery 106 is a lithium-ion battery, etc.

[0028] The head-mounted device 100 may be a helmet or glasses of the type of extended reality (XR), such as augmented reality (AR), virtual reality (VR), or mixed reality (MR), and this disclosure does not limit this type. Figure 1 The components shown are merely illustrative. The head-mounted device 100 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The head-mounted device 100 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0029] In this embodiment, the first memory 102 of the head-mounted device 100 is used to store computer instructions, and the first processor 101 is used to retrieve the computer instructions from the memory to execute the head-mounted device's photographing method. Those skilled in the art can design the instructions according to the scheme disclosed in this invention. How the instructions control the first processor 101 to operate is well known in the art and will not be described in detail here.

[0030] In one embodiment of this disclosure, such as Figure 1 As shown, the control device 200 may include a second processor 201, a second memory 202, a second communication device 203, a display device 204, a second camera device 205, etc. Furthermore, the control device 200 may also include a second power management chip 206 and a second battery 207, etc.

[0031] The second processor 201 can be any of various processors, such as CPU, GPU, NPU, ISP, and DSP. The second memory 202 can store the underlying software, system software, application software, and data required for the operation of the control device 200. The second memory 202 can include various forms of memory, such as ROM, RAM, and Flash, where RAM can be DDR. The second communication device 203 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, or 5G communication device. The second display device 204 can be a touchscreen. The second camera device 205 can be a single camera or multiple cameras. The second power management chip 206 is used to manage the power supply of the input control device 200 and can also manage the second battery 207 to ensure high utilization efficiency. The second battery 207 is a lithium-ion battery, etc.

[0032] The control device 200 can be an Android device, such as a mobile phone, tablet computer, PDA, or a control device PUCK adapted to the head-mounted device 100. Figure 1 The components shown are merely illustrative. The control device 200 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The control device 200 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0033] Taking the control device 200 as a PUCK as an example, the PUCK includes a second processor 201, a second memory 202, a second communication device 203, and a second camera device 205. The second processor 201 is an image signal processor (ISP), which includes an image decompression module, a Bayer Processing Segment (BPS) module, and an image processing engine (IPE) module. The PUCK can be used to control the head-mounted device 100, such as AR glasses, and can also be used to receive raw images sent by the head-mounted device 100 and process the raw images through the image signal processor (ISP) to obtain captured images.

[0034] In this embodiment, the second memory 202 of the control device 200 is used to store computer instructions, and the second processor 201 is used to retrieve the computer instructions from the second memory 202 to execute the imaging method of the head-mounted device applied to the control device. Those skilled in the art can design the instructions according to the scheme disclosed in this invention. How the instructions control the operation of the second processor 202 is well known in the art and will not be described in detail here.

[0035] It should be understood that, despite Figure 1 Only one head-mounted device 100 and one control device 200 are shown, but this does not mean that the number of each is limited. The head-mounted device photography system 10 may contain multiple head-mounted devices 100 and multiple control devices 200.

[0036] <Method Example 1> like Figure 2 As shown, this disclosure provides a method for taking pictures using a head-mounted device. The method is applied to the head-mounted device and includes the following steps S2100 to S2500.

[0037] Step S2100: Upon receiving the photo-taking instruction, the first raw image group is acquired.

[0038] In one embodiment of this disclosure, the photo-taking command in step S2100 can be a photo-taking command input by the user to the head-mounted device. Specifically, when the camera application of the head-mounted device is open, the user can input a photo-taking command through the photo-taking controls provided by the camera application. For example, the photo-taking command can be a touch input by the user to the photo-taking controls of the camera application, and the touch input can be a click input.

[0039] In another embodiment of this disclosure, the photo-taking command in step S2100 can also be a photo-taking command sent by the user to the head-mounted device via the control device. Specifically, the head-mounted device and the control device are connected by communication. The control device has a control application installed on it, and the control device controls the head-mounted device based on the control application. When the control application on the control device is running, the user inputs the photo-taking command through the photo-taking controls provided by the control application. The control device sends the photo-taking command to the head-mounted device via the control application based on the communication connection with the head-mounted device.

[0040] When the head-mounted device receives a photo-taking command, it controls the first camera device to capture at least one frame of raw image using the default acquisition method, thus obtaining a first raw image set. The first raw image set can be a set of raw images directly captured by the first camera device, or it can be a set of raw images obtained by processing the directly captured raw images through an image front-end.

[0041] It should be noted that any original image mentioned in this disclosure specifically refers to an image in raw format, also known as a raw image.

[0042] After obtaining the first set of raw images, the head-mounted device performs the following step S2200.

[0043] Step S2200: Determine the current environmental scene category based on the first original image group.

[0044] The current environment scene categories include high dynamic range and low dynamic range.

[0045] Specifically, the high dynamic range (HDR) category refers to the category where the brightness difference between the brightest and darkest parts of an image is greater than a preset brightness difference value. Conversely, the low dynamic range (LVR) category refers to the category where the brightness difference between the brightest and darkest parts of an image is less than or equal to a preset brightness difference value. The preset brightness difference value is the maximum brightness difference between the brightest and darkest parts of an image when the image closely approximates the realistic effect seen by the human eye; it can be determined based on experience or experimentation.

[0046] In one embodiment of this disclosure, an Automatic Scene Detection (ASD) algorithm can be used to identify the current environmental scene. Specifically, for any frame of the original image in the first original image group, the original image is input into the ASD algorithm, which identifies the environmental scene corresponding to the original image. Based on the environmental scene corresponding to each frame of the original image in the first original image group, the current environmental scene is determined. In one example, the environmental scene with the highest proportion among the environmental scenes corresponding to the original images in the first original image group is determined as the current environmental scene.

[0047] Furthermore, the head-mounted device pre-stores the correspondence between environmental scenes and environmental scene categories. After identifying the current environmental scene, the current environmental scene category is determined by combining the aforementioned correspondence. In one example, environmental scenes corresponding to the high dynamic range category include backlit scenes, night scenes, and stage performance scenes. Scenes corresponding to the low dynamic range category include landscape scenes and portrait scenes.

[0048] It should be noted that the computing power corresponding to step S2200 is small, and the power consumption of the head-mounted device is low.

[0049] Step S2300: Determine the original image acquisition method and the first image preprocessing method corresponding to the current environmental scene category.

[0050] In this embodiment, different environmental scene categories correspond to different original image acquisition methods and first image preprocessing methods. Therefore, in one embodiment of this disclosure, step S2300 is specifically implemented through the following steps S2310 and S2320.

[0051] Step S2310: When the current environment scene category is high dynamic range category, determine that the original image acquisition method is to acquire at least three frames of original images with different exposure values ​​and the first image preprocessing method is high dynamic range processing.

[0052] In this embodiment of the disclosure, when the current environmental scene category is high dynamic range, it is indicated that the first camera device is controlled to acquire original images that cannot capture a realistic effect close to what the human eye sees, according to the default acquisition method. In this case, the original image acquisition method is determined to be acquiring at least three frames of original images with different exposure values ​​(EV), and the first image preprocessing method is high dynamic range (HDR) processing. The aforementioned at least three frames are typically three to five frames.

[0053] In the case of high dynamic range processing as the first image preprocessing method, the first image preprocessing method includes, in sequence, a first alignment process, a ghosting removal process, and an HDR fusion process. Specifically, the first alignment process involves using the medium-exposure original image from at least three frames of original images with different exposure values ​​in the second original image group as a reference image. The long-exposure original images and short-exposure original images (excluding the reference image) from the at least three frames of original images with different exposure values ​​are aligned to the reference image using Bayer domain sub-pixel alignment (to avoid detail loss caused by initial mosaic removal). The ghosting removal process involves detecting moving regions in each frame of the original image obtained after the first alignment process based on optical flow or patch matching. Pixels in the moving regions are then subjected to single-frame weighting or removal processing to obtain the processed original image. The HDR fusion process involves calculating weights based on exposure value and pixel brightness in each frame of the original image obtained after the ghosting removal process (short exposures retain highlights, long exposures retain shadows), generating a 16-bit linear original image as the original image after the first preprocessing.

[0054] Step S2320: When the current environment scene category is low dynamic range category, determine that the original image acquisition method is to acquire multiple frames of original images with the same exposure value and the first image preprocessing method is multi-frame noise reduction processing.

[0055] In this embodiment of the disclosure, when the current environmental scene category is high dynamic range, the first camera device is controlled to acquire original images that closely approximate the realistic effect seen by the human eye using the default acquisition method. In this case, the original image acquisition method is determined to be acquiring multiple frames of original images with the same exposure value, and the first image preprocessing method is multi-frame noise reduction (MFNR) processing. The aforementioned multiple frames are typically 10 to 20 frames.

[0056] In the case where the first image preprocessing method is multi-frame denoising, the first image preprocessing method includes, in sequence: second alignment processing, multi-frame denoising processing, and frame fusion processing. The second alignment processing specifically involves aligning any original image in the second original image group using global motion estimation and local fine-tuning to compensate for camera shake and minor displacement. The multi-frame denoising processing specifically involves using Non-local Means (NLM) or Principal Component Analysis (PCA) to denoise each original image after the second alignment processing, preserving edge details while removing random noise, resulting in each original image after multi-frame denoising processing. The frame fusion processing specifically involves performing a weighted average (higher weight for center frames, lower weight for edge frames) on each original image after multi-frame denoising processing to generate a low-noise 14-bit original image, which serves as the original image after the first preprocessing.

[0057] Step S2400: Process the second original image group acquired according to the original image acquisition method according to the first image preprocessing method to obtain the first preprocessed original image.

[0058] It is understood that, prior to step S2400, the process includes controlling the first camera device to acquire multiple frames of original images according to the original image acquisition method determined in step S2300, thereby obtaining a second set of original images. Further, the second set of original images is processed according to the first image preprocessing method determined in step S2300, resulting in a single frame of preprocessed original image.

[0059] For step S2400 above, if the original image acquisition method is to acquire at least three original images with different exposure values ​​and the first image preprocessing method is high dynamic range processing, then a high dynamic range raw image can be generated through step S2400 above.

[0060] When the original image acquisition method is to acquire multiple frames of original images with the same exposure value and the first image preprocessing method is multi-frame noise reduction processing, a low-noise raw image can be generated through the above step S2400.

[0061] Through step S2400, the head-mounted device processes the second original image group, which includes multiple frames of original images, into a single frame of first preprocessed original image. This reduces the transmission bandwidth pressure when the head-mounted device sends images to the control device. Simultaneously, since the original images in the second original image group are acquired by the head-mounted device itself, there is no issue of image information loss due to external transmission. Therefore, the head-mounted device can perform the first image preprocessing method on the second original image group containing complete image information to obtain an accurate first preprocessed original image.

[0062] It should be noted that the computing power corresponding to step S2400 is small, and the power consumption of the head-mounted device is low.

[0063] Step S2500: The first preprocessed original image is sent to the control device that is communicatively connected to the head-mounted device.

[0064] In this process, the control device generates a photographed image based on the first preprocessed original image after receiving it.

[0065] In this embodiment, after the head-mounted device completes step S2400 to obtain the first preprocessed original image, it no longer processes the first preprocessed original image. Instead, it sends the first preprocessed original image to the control device, which then processes it into a photographed image. The photographed image is exemplified as an image in Joint Photographic Experts Group (JPEG) format. This allows the control device to perform the high-power, offline-processable photographic encoding process, effectively reducing the overall power consumption of the head-mounted device.

[0066] Based on the above, this disclosure provides a method for taking pictures using a head-mounted device. The method includes: acquiring a first set of original images upon receiving a picture-taking command; determining the current environmental scene category based on the first set of original images, where the current environmental scene category includes a high dynamic range (HDR) category and a low dynamic range (LVR) category; determining the original image acquisition method and a first image preprocessing method corresponding to the current environmental scene category; processing the second set of original images acquired using the original image acquisition method according to the first image preprocessing method to obtain a first preprocessed original image; and sending the first preprocessed original image to a control device communicatively connected to the head-mounted device. Upon receiving the first preprocessed original image, the control device generates a picture based on the first preprocessed original image. In this method, the head-mounted device performs lightweight preprocessing to obtain an accurate first preprocessed original image, while the control device performs high-power, offline-processable picture encoding processing, effectively reducing the overall power consumption of the head-mounted device while maintaining image accuracy.

[0067] In one embodiment of this disclosure, the head-mounted device photography method provided by this disclosure, before the above-described step S2400, further includes the following steps S2410 and S2420.

[0068] Step S2410: Acquire multiple frames of original images according to the original image acquisition method to obtain the third set of original images.

[0069] Step S2420: Process each frame of the original image in the third original image group according to the second image preprocessing method to obtain the second original image group. The second image preprocessing method is the image front-end processing method.

[0070] In one embodiment of this disclosure, the image front-end processing method includes at least one of: basic correction processing, shadow correction processing, and bad pixel correction processing.

[0071] In this disclosure, multiple frames of original images directly acquired using the original image acquisition method are denoted as the third original image group. After obtaining the third original image group, for each frame of the third original image group, at least one of the following image front-end processing methods is performed: basic correction processing, shadow correction processing, and bad pixel correction processing, to obtain a preprocessed original image. The preprocessed original image corresponding to each frame of the third original image group is denoted as the second original image group.

[0072] Through steps S2410 and S2420 described above, image front-end processing of the original images in the third original image group can be achieved to obtain a second original image group with high image quality. It is understood that the image front-end processing requires less computational power, thus resulting in low power consumption for the head-mounted device.

[0073] In one embodiment of this disclosure, step S2200 is specifically implemented through step S2210.

[0074] Step S2210: Determine the current environment scene category and the current image sharpness value based on the first original image group.

[0075] In this embodiment of the disclosure, after obtaining the first original image group, the current environmental scene category is determined simultaneously with the current image sharpness value. In one embodiment of the disclosure, the image sharpness value of a random frame of the original image in the first original image group can be determined as the current image sharpness value, or the maximum original image sharpness value among all frames of the original image in the first original image group can be determined as the current image sharpness value.

[0076] In one example, the image sharpness value of the original image can be determined based on an image sharpness detection algorithm.

[0077] Based on step S2210 above, step S2500 above is specifically implemented through steps S2510 to S2530 below.

[0078] Step S2510: Determine the image compression method based on the current image sharpness value.

[0079] In one embodiment of this disclosure, step S2510 is specifically implemented through the following steps S2511 and S2512.

[0080] Step S2511: If the current image sharpness value is greater than the preset sharpness value, determine that the image compression method is lossless compression.

[0081] Step S2512: If the current image sharpness value is greater than the preset sharpness value, determine that the image compression method is lossless compression.

[0082] The preset sharpness value is the maximum sharpness value of the image after decompression using visual lossless compression, where visual defects are present. When the first original image group is a text-based image group, the current image sharpness value is usually greater than the preset sharpness value. When the first original image group is an image-based image group, the current image sharpness value is usually less than or equal to the preset sharpness value.

[0083] Based on the above, if the current image sharpness value is greater than the preset sharpness value, then the image compression method is determined to be lossless compression. This ensures that the compressed image obtained after lossless compression will have no visual defects after decompression.

[0084] Correspondingly, if the current image sharpness value is less than or equal to the preset sharpness value, then the image compression method is determined to be visually lossless compression. In this way, while reducing the amount of image data, it is still possible to ensure that the compressed image obtained after lossless compression is visually intact after decompression.

[0085] Step S2520: Compress the first preprocessed original image into a compressed image according to the image compression method.

[0086] Specifically, if the image compression method is lossless compression, then the original image after the first preprocessing is compressed using lossless compression. Similarly, if the image compression method is visually lossless compression, then the original image after the first preprocessing is compressed using visually lossless compression. The compressed image is denoted as the compressed image.

[0087] Step S2530: The compressed image is sent to a control device that is communicatively connected to the head-mounted device.

[0088] Since the compression ratio of lossless compression is usually 30%-40%, while the compression ratio of visual lossless compression is usually 60%-80%, the compression method can be dynamically selected according to the current image sharpness value through the above steps S2510 to S2530. In this way, while ensuring image quality, the amount of data of compressed image can be effectively reduced, thereby effectively reducing the pressure on external transmission bandwidth.

[0089] <Method Example 2> like Figure 3As shown, this disclosure also provides a method for taking pictures using a head-mounted device applied to a control device, including the following steps S3100 and S3200.

[0090] Step S3100: Receive the first preprocessed raw image sent by the head-mounted device.

[0091] Corresponding to step S2500 above, the control device can receive the first preprocessed original image sent by the head-mounted device.

[0092] Step S3200: Perform Bayer processing and image back-end processing sequentially on the first preprocessed original image to obtain the captured image.

[0093] In this embodiment of the disclosure, after receiving the first pre-processed original image sent by the head-mounted device, the control device sequentially performs Bayer Processing Segment (BPS) and Image Processing Engine (IPE) processing on the first pre-processed original image to complete the high-power, offline-processable image encoding process and obtain the captured image.

[0094] In one embodiment of this disclosure, step S3100 is specifically implemented through the following steps S3110 and S3120.

[0095] Step S3110: Receive the compressed image obtained by compressing the original image after the first preprocessing, sent by the head-mounted device.

[0096] Step S3120: Decompress the compressed image to obtain the first preprocessed original image.

[0097] Corresponding to step S2530 above, the control device can receive a compressed image obtained by compressing the first preprocessed original image sent by the head-mounted device. After obtaining the compressed image, the control device parses the compressed image to determine the compression method corresponding to the compressed image. Then, the control device uses the decompression method corresponding to the determined compression method to decompress the compressed image to obtain the first preprocessed original image.

[0098] <Example> Based on the above, taking AR glasses as the head-mounted device and PUCK as the control device as an example, the photographing method of the head-mounted device provided in this disclosure is explained as follows: Figure 4 As shown, the method includes the following steps.

[0099] In step S401, after the AR glasses start the camera application and receive the photo-taking command, the first camera device outputs a serial data stream representing multiple frames of original images to the camera serial interface decoder (CSID) module through the MIPI interface.

[0100] In step S402, the CSID module receives and decodes the serial data stream transmitted from the camera through the MIPI interface to obtain the original image in an image data format that can be used for subsequent image processing, and transmits it to the IFE module on the head-mounted device side. The IFE module on the head-mounted device side performs image front-end processing (Pre-raw) on the received original image to obtain the first original image group, and transmits it to the statistics module (Stats).

[0101] In step S403, the Stats module analyzes the first group of original images to obtain the current environment scene category and the current image sharpness value. It further determines the image compression method based on the image sharpness value, determines the subsequent original image acquisition method and the first image preprocessing method based on the current environment scene category, and further controls the acquisition of multiple frames of original images according to the original image acquisition method.

[0102] In step S404, the camera acquires multiple frames of original images using the original image acquisition method. The multiple frames of original images acquired using the original image acquisition method are processed into a second set of original images after passing through the CSID module and the IFE module on the head-mounted device side.

[0103] In step S405, the HDR / MFNR module processes the second set of original images acquired according to the original image acquisition method according to the first image preprocessing method to obtain the first preprocessed original image, and then transmits it to the Compress module.

[0104] In step S406, the Compress module compresses the original image after the first preprocessing according to the image compression method determined in step S403 above, obtains the compressed image, and stores it in the DDR on the head-mounted device side. The first communication device sends the compressed image stored in the DDR to PUCK.

[0105] In step S501, the second communication device on the PUCK side stores the compressed image sent by the head-mounted device in the DDR on the PUCK side.

[0106] In step S502, the decompression module on the PUCK side reads the compressed image from the DDR on the PUCK side and decompresses it into the first pre-processed original image according to the corresponding decompression method, and further stores the first pre-processed original image in the DDR on the PUCK side.

[0107] In step S503, the BPS module on the PUCK side performs Bayer processing on the first preprocessed original image to obtain the first preprocessed original image after Bayer processing and continues to store it in the DDR on the PUCK side.

[0108] In step S504, the IPE module on the PUCK side performs image back-end processing on the first preprocessed original image after processing by the Bayer processing section, obtaining the first preprocessed original image after image back-end processing, i.e., the captured image JPEG, and stores it in the DDR on the PUCK side. In this way, the JPEG image can be extracted from the DDR on the PUCK side.

[0109] Furthermore, it should be noted that when the head-mounted device includes a first display device, such as Figure 4 As shown, this disclosure does not modify the preview process of the head-mounted device. The preview process is as follows: the first raw image group output by the IFE on the head-mounted device side is sequentially passed through the DDR on the head-mounted device side, the IPE module on the head-mounted device side, and the DDR on the head-mounted device side to become the preview image Preivew.

[0110] <Device Example 1> This disclosure also provides a camera 500 for a head-mounted device, wherein the camera 500 is used in a head-mounted device, such as Figure 5 As shown, the imaging device 500 of the head-mounted device includes: The acquisition module 510 is used to acquire a first group of raw images when a photo-taking command is received; The first determining module 520 is used to determine the current environment scene category based on the first original image group, wherein the current environment scene category includes a high dynamic range category and a low dynamic range category; The second determining module 530 is used to determine the original image acquisition method and the first image preprocessing method corresponding to the current environmental scene category; The first processing module 540 is used to process the second original image group acquired according to the original image acquisition method according to the first image preprocessing method to obtain the first preprocessed original image. The sending module 550 is used to send the first preprocessed original image to a control device that is communicatively connected to the head-mounted device. Upon receiving the first preprocessed original image, the control device generates a photographed image based on the first preprocessed original image.

[0111] In one embodiment of this disclosure, the second determining module 530 is specifically used for: When the current environment scene category is high dynamic range category, the original image acquisition method is determined to be acquiring at least three frames of original images with different exposure values ​​and the first image preprocessing method is high dynamic range processing; When the current environment scene category is low dynamic range, the original image acquisition method is determined to be acquiring multiple frames of original images with the same exposure value, and the first image preprocessing method is multi-frame noise reduction processing.

[0112] In one embodiment of this disclosure, the head-mounted camera device 500 provided in this disclosure further includes: The third processing module is used to acquire multiple frames of original images according to the original image acquisition method to obtain the third original image group; According to the second image preprocessing method, each frame of the original image in the third original image group is processed to obtain the second original image group. The second image preprocessing is an image front-end processing method.

[0113] In one embodiment of this disclosure, the first determining module 520 is specifically used to determine the current environmental scene category and the current image sharpness value based on the first original image group. In this embodiment of the disclosure, the sending module 550 is specifically used to determine the image compression method based on the current image sharpness value; The first preprocessed original image is compressed into a compressed image according to the image compression method described above; The compressed image is sent to a control device that is communicatively connected to the head-mounted device.

[0114] <Device Embodiment Two> When the camera device 500 of the head-mounted device is used as a control device, such as Figure 6 As shown, the imaging device 500 of the head-mounted device includes: The receiving module 560 is used to receive the first preprocessed original image sent by the head-mounted device; The second processing module 570 is used to sequentially perform Bayer processing and image back-end processing on the first preprocessed original image to obtain a photographed image.

[0115] In one embodiment of this disclosure, the receiving module 560 is specifically used to receive a compressed image obtained by compressing the first preprocessed original image sent by the head-mounted device. Decompress the compressed image to obtain the first preprocessed original image.

[0116] <Equipment Example 1> This disclosure also provides an electronic device, which is a head-mounted device, comprising a first processor and a first memory, wherein the first memory is used to store computer instructions, and the first processor is used to call the computer instructions from the first memory to perform the method as described in any one of the above method embodiments.

[0117] Alternatively, the electronic device may include the camera device 500 of any of the head-mounted devices described in Embodiment 1 above.

[0118] <Equipment Example 2> This disclosure also provides another electronic device, which is a control device, comprising a second processor and a second memory, wherein the second memory is used to store computer instructions, and the second processor is used to retrieve the computer instructions from the second memory to perform the method described in any one of the two above-described method embodiments.

[0119] Alternatively, the electronic device may include the camera device 500 of any of the head-mounted devices described in Embodiment 2 above.

[0120] <System Equipment Implementation Example> This disclosure also provides a head-mounted device photography system, including the head-mounted device as described in the first device embodiment above and the control device as described in the second device embodiment above.

[0121] <Example of a Media Device> This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the above-described method embodiment one or method embodiment two.

[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0123] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0124] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0125] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0126] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, can execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0127] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0128] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0129] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0131] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for taking photos using a head-mounted device, characterized in that, Applied to head-mounted devices, the method includes: Upon receiving a photo-taking command, the first set of raw images is captured; Based on the first original image group, the current environment scene category is determined, and the current environment scene category includes a high dynamic range category and a low dynamic range category; Determine the original image acquisition method and the first image preprocessing method corresponding to the current environmental scene category; According to the first image preprocessing method, the second original image group acquired according to the original image acquisition method is processed to obtain the first preprocessed original image; The first preprocessed original image is sent to a control device that is communicatively connected to the head-mounted device. Upon receiving the first preprocessed original image, the control device generates a photographed image based on the first preprocessed original image.

2. The method according to claim 1, characterized in that, The determination of the original image acquisition method and the first image preprocessing method corresponding to the current environmental scene category includes: When the current environment scene category is high dynamic range category, the original image acquisition method is determined to be acquiring at least three frames of original images with different exposure values ​​and the first image preprocessing method is high dynamic range processing; When the current environment scene category is low dynamic range, the original image acquisition method is determined to be acquiring multiple frames of original images with the same exposure value, and the first image preprocessing method is multi-frame noise reduction processing.

3. The method according to claim 1, characterized in that, Before processing the second set of original images acquired according to the original image acquisition method according to the first image preprocessing method to obtain the first preprocessed original image, the method further includes: Multiple frames of original images were acquired using the original image acquisition method to obtain the third set of original images; According to the second image preprocessing method, each frame of the original image in the third original image group is processed to obtain the second original image group. The second image preprocessing method is the image front-end processing method.

4. The method according to claim 1, characterized in that, The step of determining the current environmental scene category based on the first original image group includes: Based on the first original image group, the current environmental scene category and the current image sharpness value are determined respectively; Sending the first preprocessed original image to a control device communicatively connected to the head-mounted device includes: The image compression method is determined based on the current image sharpness value; The first preprocessed original image is compressed into a compressed image according to the image compression method described above; The compressed image is sent to a control device that is communicatively connected to the head-mounted device.

5. The method according to claim 4, characterized in that, The step of determining the image compression method based on the current image sharpness value includes: If the current image sharpness value is greater than the preset sharpness value, the image compression method is determined to be lossless compression. If the current image sharpness value is less than or equal to a preset sharpness value, the image compression method is determined to be visual lossless compression.

6. A method for taking photos using a head-mounted device, characterized in that, The method is applied to a control device communicatively connected to a head-mounted device, and the method includes: Receive the first preprocessed raw image sent by the head-mounted device; The Bayer processing stage and the image back-end processing are sequentially performed on the first preprocessed original image to obtain the photographed image.

7. The method according to claim 6, characterized in that, The receiving of the first preprocessed original image sent by the head-mounted device includes: Receive a compressed image obtained by compressing the original image after the first preprocessing, sent by the head-mounted device; Decompress the compressed image to obtain the first preprocessed original image.

8. An electronic device, characterized in that, When the electronic device is a head-mounted device, the electronic device includes a first processor and a first memory, the first memory being used to store computer instructions, and the first processor being used to retrieve the computer instructions from the first memory to perform the method as described in any one of claims 1-5; When the electronic device is a control device, the electronic device includes a second processor and a second memory, the second memory being used to store computer instructions, and the second processor being used to retrieve the computer instructions from the second memory to execute the method as described in claim 6 or 7.

9. A head-mounted camera system, characterized in that, It includes the head-mounted device as described in claim 8 and the control device as described in claim 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images