Image processing methods, apparatus, equipment and systems

Abstract

This disclosure provides an image processing method, apparatus, device, and system applied to a head-mounted display device. The head-mounted display device includes a camera device, which includes an image sensor. The method includes: receiving a photo-taking command; generating an original image in response to the photo-taking command; wherein the original image is an image output by the image sensor; and sending the original image to a control device for image processing to generate a photographed image.

Description

Technical Field

[0001] This disclosure relates to the field of head-mounted display technology, and more specifically, to an image processing method, an image processing apparatus, an electronic device, and an image processing system. Background Technology

[0002] Power consumption is a key metric for the success of Augmented Reality (AR) products, impacting overall product design. Effective power consumption allows for the use of low-capacity batteries, thus reducing overall weight. However, the camera, being a high-power component in AR products, can draw up to 1A of current during image capture, significantly increasing overall power consumption. Therefore, optimizing image capture power consumption can effectively reduce the overall power consumption of AR products. Summary of the Invention

[0003] The purpose of this disclosure is to provide an image processing method, apparatus, device, and system.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, applied to a head-mounted display device, the head-mounted display device including a camera device, the camera device including an image sensor, the method comprising:

[0005] Receive photo capture command;

[0006] In response to the photographing command, an original image is generated; wherein the original image is the image output by the image sensor;

[0007] The original image is sent to the control device so that the control device can perform image processing on the original image to generate a photographed image.

[0008] Optionally, receiving the photo-taking instruction includes:

[0009] Receive a photo-taking command input by the user for the head-mounted display device; or,

[0010] The device receives a photo-taking command sent from the control device to the head-mounted display device.

[0011] Optionally, before receiving the photo-taking command, the method further includes:

[0012] Receive the launch command to start the camera application;

[0013] In response to the launch command, the camera application is launched;

[0014] With the camera application running, a first preview image is generated.

[0015] According to a second aspect of the present disclosure, an image processing method is provided, applied to a control device, the method comprising:

[0016] Receive raw images sent by a head-mounted display device; wherein, the raw images are images output by the image sensor in the camera device of the head-mounted display device when the head-mounted display device receives a photo-taking command;

[0017] The original image is processed to generate a photographed image.

[0018] Optionally, the step of performing image processing on the original image to generate a photographed image includes:

[0019] The image signal processor of the control device performs image processing on the original image to generate the captured image;

[0020] The image signal processor includes at least a raw image processing module, an image back-end processing module, and an image encoding module.

[0021] Optionally, before receiving the raw image sent by the head-mounted display device, the method further includes:

[0022] Control the control device to enter the test image mode;

[0023] When the control device enters the test mode, the pre-stored test image is acquired through the camera serial interface decoder of the control device;

[0024] A second preview image is generated based on the test image.

[0025] According to a third aspect of the present disclosure, an image processing apparatus is provided, applied to a head-mounted display device, the head-mounted display device including a camera device, the camera device including an image sensor, the apparatus comprising:

[0026] The first receiving module is used to receive photo-taking commands;

[0027] The first generation module is used to generate an original image in response to the photo-taking command; wherein the original image is the image output by the image sensor;

[0028] The sending module is used to send the original image to the control device, so that the control device can perform image processing on the original image to generate a photographed image.

[0029] According to a fourth aspect of the present disclosure, an image processing apparatus is provided for use in a control device, the apparatus comprising:

[0030] The second receiving module is used to receive the original image sent by the head-mounted display device; wherein, the original image is the image output by the image sensor in the camera device of the head-mounted display device when the head-mounted display device receives a photo-taking command;

[0031] The second generation module is used to perform image processing on the original image to generate a photographed image.

[0032] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0033] Memory is used to store executable computer instructions;

[0034] A processor, configured to execute the image processing method according to the first aspect above, under the control of the executable computer instructions.

[0035] According to a sixth aspect of the present disclosure, an image processing system is provided, the image processing system including a head-mounted display device and a control device, the head-mounted display device and the control device being communicatively connected.

[0036] The head-mounted display device is configured to receive a photo-taking command, and in response to the photo-taking command, generate an original image and send the original image to the control device; wherein the original image is an image output by the image sensor in the camera device of the head-mounted display device;

[0037] The control device is used to receive the original image sent by the head-mounted display device, and to perform image processing on the original image to generate a photographed image.

[0038] One beneficial effect of this disclosure is that, upon receiving a photo-taking command, the head-mounted display device generates an original image in response to the command. This original image is the image output by the image sensor. The original image is then sent to a control device, which performs image processing on the original image to generate a photographed image. In other words, this improves upon the high-power, offline-processing photo-taking workflow of head-mounted display devices by sending the original image output by the image sensor to the control device for image processing to generate a photographed image, effectively reducing the power consumption of the head-mounted display device.

[0039] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0041] Figure 1 This is a schematic diagram of the hardware configuration of an image processing system according to an embodiment of the present disclosure;

[0042] Figure 2 This is a schematic flowchart of an image processing method according to an embodiment of the present disclosure;

[0043] Figure 3 This is a schematic flowchart of an image processing method according to another embodiment of the present disclosure;

[0044] Figure 4 This is a schematic diagram of the interaction process between a head-mounted display device and a control device, based on an example.

[0045] Figure 5 This is a schematic block diagram of an image processing apparatus according to embodiments of the present disclosure;

[0046] Figure 6 This is a schematic block diagram of an image processing apparatus according to another embodiment of the present disclosure;

[0047] Figure 7 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure;

[0048] Figure 8 This is a schematic block diagram of an image processing system according to an embodiment of the present disclosure. Detailed Implementation

[0049] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.

[0050] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0051] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0052] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0053] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0054] <Hardware Configuration>

[0055] Figure 1 This is a schematic diagram of the hardware configuration of an image processing system that can be used to implement one embodiment. Figure 1 A head-mounted display device 100 and a control device 200 are shown. The head-mounted display device 100 and the control device 200 are communicatively connected.

[0056] In one embodiment, such as Figure 1 As shown, the head-mounted display device 100 may include at least a processor 101, a memory 102, a communication device 103, and a camera device 104. Furthermore, the head-mounted display device 100 may also include a power management chip 105 and a battery 106, etc. It should be noted that the head-mounted display device 100 may or may not have a display device (not shown in the figure), wherein the display device may be a screen.

[0057] The processor 101 can be any type of processor. The memory 102 stores the underlying software, system software, application software, data, etc., required for the operation of the head-mounted display device 100. The memory 102 can include various forms of memory, such as ROM, RAM, Flash, etc., where RAM can be Double Data Rate SDRAM (DDR). The communication device 103 can include, for example, WiFi communication devices, Bluetooth communication devices, 3G, 4G, and 5G communication devices. The camera device 104 can be a single camera or multiple cameras. The power management chip 105 manages the power input to the head-mounted display device 100 and also manages the battery 106 to ensure high utilization efficiency. The battery 106 is a lithium-ion battery, etc.

[0058] The head-mounted display device 100 may be AR (Augmented Reality) glasses or MR (Mixed Reality) glasses, etc., and this disclosure does not limit it. Figure 1 The components shown are merely illustrative. The head-mounted display device 100 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The head-mounted display device 100 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0059] In this embodiment, the memory 102 of the head-mounted display device 100 is used to store program instructions that control the processor 101 to perform image processing methods. Those skilled in the art can design these instructions based on the disclosed scheme of this invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0060] In one embodiment, such as Figure 1 As shown, the control device 200 may include a processor 201, a memory 202, a communication device 203, a display device 204, a camera device 205, etc. Furthermore, the control device 200 may also include a power management chip 206 and a battery 207, etc.

[0061] The processor 201 can be any type of processor. The memory 202 can store the underlying software, system software, application software, data, etc., required for the operation of the control device 200. The memory 202 can include various forms of memory, such as ROM, RAM, Flash, etc., where RAM can be DDR. The communication device 203 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, and 5G communication device. The display device 204 can be a touchscreen. The camera device 205 can be a single camera or multiple cameras. The power management chip 206 is used to manage the power supply of the input control device 200 and can also manage the battery 207 to ensure maximum utilization efficiency. The battery 207 is a lithium-ion battery, etc.

[0062] The control device 200 can be an Android device, such as a mobile phone, tablet, handheld computer, PUCK, etc. Figure 1 The components shown are merely illustrative. The control device 200 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The control device 200 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0063] Taking the control device 200 as a PUCK as an example, the PUCK includes a processor 201, a memory 202, a communication device 203, and a camera device 205. The camera device 205 includes an image signal processor (ISP), which comprises a Bayer Processing Segment (BPS), an Image Front End (IFE), an Image Processing Engine (IPE), and an image encoding module. The PUCK can be used to control head-mounted display devices such as AR glasses, and can also be used to receive raw images sent by the head-mounted display device and process the raw images through the image signal processor (ISP) to obtain captured images.

[0064] In this embodiment, the memory 202 of the control device 200 is used to store program instructions that control the processor 201 to perform an image processing method. Those skilled in the art can design these instructions based on the disclosed scheme of this invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0065] It should be understood that, despite Figure 1 Only one head-mounted display device 100 and one control device 200 are shown, but this does not mean that the number of each is limited. The image processing system may contain multiple head-mounted display devices 100 and multiple control devices 200.

[0066] In the above description, those skilled in the art can design instructions based on the solutions provided in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0067] <Method Example 1>

[0068] Figure 2 An image processing method according to an embodiment of this disclosure is shown, which can be generated by... Figure 1 The illustrated head-mounted display device includes a camera device, which includes an image sensor. Figure 2 As shown, the image processing method of this embodiment may include the following steps S2100 to S2300:

[0069] Step S2100: Receive the photo-taking command.

[0070] Optionally, this step S2100 of receiving the photo-taking command may further include: receiving a photo-taking command input by the user for the head-mounted display device.

[0071] Specifically, when the camera application of the head-mounted display device is open, the user can input a photo-taking command through the photo-taking controls provided by the camera application. For example, this photo-taking command can be a touch input from the user to the photo-taking controls of the camera application, and this touch input can be a tap input.

[0072] Optionally, step S2100, receiving the photo-taking instruction, may further include: receiving a photo-taking instruction sent by the control device to the head-mounted display device.

[0073] It should be noted that, on the one hand, there is a communication connection between the head-mounted display device and the control device, which can be a WiFi connection. On the other hand, the control device usually has a control application installed, which allows it to control the head-mounted display device.

[0074] Specifically, when the control application in the control device is running, the control device can send a photo-taking command to the head-mounted display device through the control application based on the communication connection between the control device and the head-mounted display device.

[0075] In an optional embodiment, before performing step S2100 to receive the photo-taking instruction, the image processing method of this embodiment further includes step S4100: receiving a startup instruction to launch a camera application; launching the camera application in response to the startup instruction; and generating a first preview image when the camera application is in the launched state.

[0076] The imaging device also includes an image signal processor (ISP). The image signal processor (ISP) includes at least an image front-end processing module (IFE), an image back-end processing module (IPE), and an image encoding module.

[0077] The aforementioned front-end processing module IFE is used for image color correction, downsampling, de-mosaicing, and statistical 3A data processing. It should be noted that the image processed by the IFE module is in YUV format, where Y represents luminance, and U and V represent chrominance.

[0078] The aforementioned image back-end processing module IPE is used to perform noise reduction, cropping, color processing, and detail enhancement on images.

[0079] The image encoding module described above is used to convert the format of the image. The image encoding module can be a JPEG (Joint Photographic Experts Group) generation module, which means that the image format after processing by the image encoding module is JPEG.

[0080] Specifically, the head-mounted display device provides a camera application, which users can launch by inputting a launch command through the camera application icon provided by the head-mounted display device. For example, this launch command can be a touch input from the user to the camera application icon, and this touch input can be a click input.

[0081] Reference Figure 4 For head-mounted display devices, when the camera application is active, the image sensor outputs a MIPI (Mobile Industry Processor Interface) data stream, which is received by the Camera Serial Interface Decoder (CSID). Upon receiving the MIPI data stream, the CSID sends it to the image front-end processing module (IFE). The IFE processes the MIPI data stream and stores it in its local memory (DDR). The image back-end processing module (IPE) retrieves the processed MIPI data stream from the local DDR, processes it, and stores it in its local DDR. The JPEG generation module retrieves the processed MIPI data stream from the local DDR and processes it... Figure 4 (Not shown in the image) Generate a preview image (Preview).

[0082] In practice, the head-mounted display device can first execute step S4100 to control the camera application to be turned on and generate a preview image in real time, and then execute step S2100 to receive the photo-taking command.

[0083] After S2100 receives the photo-taking command, the following steps are performed:

[0084] Step S2200: In response to the photographing command, generate the original image.

[0085] The original image is the image output by the image sensor, and is usually referred to as a RAW image.

[0086] In this embodiment, after receiving a photo-taking command, the head-mounted display device can respond to the command by generating an original image through an image sensor.

[0087] Reference Figure 4 When the head-mounted display receives a photo-taking command, it generates a RAW image through the image sensor and stores it in the local DDR.

[0088] After executing the above step S2200 in response to the photographing command and generating the original image, proceed to:

[0089] Step S2300: The original image is sent to the control device so that the control device can perform image processing on the original image to generate a photographed image.

[0090] In this embodiment, after the image sensor generates the original image, the head-mounted display device first stores the original image in the local DDR, and then sends the original image to the control device for image processing to generate a photographed image based on the communication connection established with the control device, such as a WiFi connection.

[0091] In practice, after receiving the original image, the control device will process the original image through its image signal processor (ISP) to generate a photographed image.

[0092] The image signal processor (ISP) includes a raw image processing module (BPS), an image back-end processing module (IPE), and an image encoding module.

[0093] The aforementioned raw image processing module, BPS (or RAW image processing module), is used to perform bad pixel removal, phase detection autofocus, depixelation, downsampling, HDR processing, and Bayer hybrid noise reduction on the raw image. It should be noted that the image processed by the raw image processing module is in YUV format.

[0094] The aforementioned image back-end processing module IPE is used to perform noise reduction, cropping, color processing, and detail enhancement on images.

[0095] The image encoding module described above is used to convert the format of the image. The image encoding module can be a JPEG generation module, that is, the image format after processing by the image encoding module is JPEG.

[0096] Reference Figure 4 The control device receives RAW images sent by the head-mounted display device and stores them in the local DDR. The raw image processing module (BPS) obtains the RAW image from the local DDR, processes it, and then stores it in the local DDR. The image back-end processing module (IPE) obtains the processed RAW image from the local DDR, processes it, and then stores it in the local DDR. The JPEG generation module obtains the processed raw image from the local DDR and processes it (not shown in the figure) to generate the captured JPEG image.

[0097] It should be noted that since the camera application of the control device is in a closed state, the control device can first enter a test image mode before receiving the original image sent by the head-mounted display device. In test image mode, the control device obtains a pre-stored test image through the camera serial interface decoder CSID, and generates a second preview image based on the test image. The image size of this test image is the same as that of the original image.

[0098] In layman's terms, the control device generates a test image as its preview image (which may not be displayed). At this time, it waits for a photo-taking event. Receiving the original image sent by the head-mounted display device is equivalent to triggering a photo-taking event, and the received original image can then be processed.

[0099] The image signal processor (ISP) of the control device also includes an image back-end processing module (IPE).

[0100] Reference Figure 4 Before receiving the RAW image from the head-mounted display device, the control device can first enter a test image mode. In this mode, the control device acquires the test image data stream through its camera serial interface decoder (CSID). Upon receiving the test image data stream, the CSID sends it to the image front-end processing module (IFE). The IFE processes the test image data stream and stores it in the local DDR. The image back-end processing module (IPE) retrieves the processed test image data stream from the local DDR, processes it, and stores it in the local DDR. The JPEG generation module retrieves the processed MIPI data stream from the local DDR and processes it (not shown in the figure) to generate a preview image. It should be noted that the control device generates the preview image based on the test image and typically discards the generated preview image.

[0101] According to embodiments of this disclosure, upon receiving a photo-taking command, the head-mounted display device generates an original image in response to the command. This original image is the image output by the image sensor. The original image is then sent to a control device, which performs image processing on the original image to generate a photographed image. In other words, for the high-power, offline-processing photo-taking process in the head-mounted display device, by sending the original image output by the image sensor at the head-mounted display device to the control device for image processing to generate a photographed image, the power consumption of the head-mounted display device is effectively reduced.

[0102] <Method Example 2>

[0103] Figure 3An image processing method according to an embodiment of this disclosure is shown, which can be generated by... Figure 1 The control equipment shown is implemented. For example... Figure 3 As shown, the image processing method of this embodiment may include the following steps S3100 to S3200:

[0104] Step S3100: Receive the raw image sent by the head-mounted display device.

[0105] The original image is the image output by the image sensor in the camera device of the head-mounted display device when the head-mounted display device receives a photo-taking command.

[0106] After performing the above step S3100 to receive the raw image sent by the head-mounted display device, proceed to:

[0107] Step S3200: Perform image processing on the original image to generate a photographed image.

[0108] In an optional embodiment, step S3200, which involves image processing of the original image to generate a photographed image, may further include: performing image processing on the original image using the image signal processor of the control device to generate the photographed image.

[0109] The image signal processor includes at least a raw image processing module (BPS), an image back-end processing module (IPE), and an image encoding module.

[0110] Reference Figure 4 The control device receives RAW images sent by the head-mounted display device and stores them in the local DDR. The raw image processing module (BPS) obtains the RAW image from the local DDR, processes it, and then stores it in the local DDR. The image back-end processing module (IPE) obtains the processed RAW image from the local DDR, processes it, and then stores it in the local DDR. The JPEG generation module obtains the processed RAW image from the local DDR and processes the processed raw image (not shown in the figure) to generate the captured JPEG image.

[0111] In an optional embodiment, before performing step S3100 above to receive the original image sent by the head-mounted display device, the image processing method of this embodiment further includes: controlling the control device to enter a test image mode; receiving a pre-stored test image through the camera serial interface of the control device when the control device enters the test mode; and generating a second preview image based on the test image.

[0112] The test image has the same image size as the original image.

[0113] It should be noted that since the camera application of the control device is in a closed state, the control device can first enter the test image mode before receiving the original image sent by the head-mounted display device. In the test image mode, the control device can obtain the pre-stored test image through the camera serial interface decoder CSID and generate a second preview image based on the test image.

[0114] In layman's terms, the control device generates a test image as its preview image (which may not be displayed). At this time, it waits for a photo-taking event. Receiving the original image sent by the head-mounted display device is equivalent to triggering a photo-taking event, and the received original image can then be processed.

[0115] The image signal processor (ISP) of the control device also includes an image back-end processing module (IFE).

[0116] Reference Figure 4 Before receiving the RAW image from the head-mounted display device, the control device can first enter a test image mode. In this mode, the control device acquires the test image data stream through its camera serial interface decoder (CSID). Upon receiving the test image data stream, the CSID sends it to the image front-end processing module (IFE). The IFE processes the test image data stream and stores it in the local DDR. The image back-end processing module (IPE) retrieves the processed test image data stream from the local DDR, processes it, and stores it in the local DDR. The JPEG generation module retrieves the processed MIPI data stream from the local DDR and processes it (not shown in the figure) to generate a preview image. It should be noted that the control device generates the preview image based on the test image and typically discards the generated preview image.

[0117] According to embodiments of this disclosure, when a head-mounted display device receives a photo-taking command, it generates an original image in response to the command. This original image is the image output by the image sensor. The original image is then sent to a control device, whereby the control device processes the original image to generate a photographed image. In other words, by using a high-power, offline-processing method for photo-taking—sending the original image output by the image sensor at the head-mounted display device to the control device for image processing to generate a photographed image—the power consumption of the head-mounted display device is effectively reduced.

[0118] <Example>

[0119] Next, refer to Figure 4Taking AR glasses as the head-mounted display device and PUCK as the control device as an example, an example image processing method is shown. In this example, the image processing method may further include:

[0120] In step S401, after the AR glasses launch the camera application, the image sensor outputs a MIPI data stream.

[0121] In step S402, the AR glasses receive the MIPI data stream via the CSID and send it to the IFE module. The IFE module processes the MIPI data stream and stores it in the local DDR. The IPE module retrieves the processed MIPI data stream from the local DDR, processes it, and stores it in the local DDR. The JPEG generation module retrieves the processed MIPI data stream from the local DDR, processes it, and generates the first preview image (Preview).

[0122] Step S403: The AR glasses receive the photo-taking command.

[0123] In step S404, the AR glasses respond to the photo-taking command by outputting a RAW image through the image sensor and storing the RAW image in the local DDR.

[0124] In step S405, the AR glasses acquire a RAW image from the local DDR and send the RAW image to the PUCK via a WiFi connection with the PUCK.

[0125] In step S501, before receiving the RAW image, PUCK obtains the test image data stream via CSID and sends it to the IFE module. The IFE module processes the test image data stream and stores it in the local DDR. The IPE module obtains the processed test image data stream from the local DDR, processes it, and stores it in the local DDR. The JPEG generation module obtains the processed test image data stream from the local DDR, processes it, and generates a second preview image (which is then discarded).

[0126] In step S502, PUCK receives the RAW image sent by the AR glasses and stores it in the local DDR. The BPS module obtains the RAW image from the local DDR, processes the obtained RAW image, and stores it in the local DDR. The IPE module obtains the processed RAW image from the local DDR, processes the obtained RAW image, and stores it in the local DDR. The JPEG generation module obtains the processed RAW image from the local DDR and generates a captured image from the obtained processed RAW image.

[0127] In this example, by sending the RAW image from the AR glasses to the PUCK image signal processor for processing and image encoding, the power consumption of the AR glasses is effectively reduced.

[0128] <Device Example 1>

[0129] Figure 5 This is a schematic diagram of an image processing apparatus according to one embodiment, applied to a head-mounted display device, the head-mounted display device including a camera device, the camera device including an image sensor. (Refer to...) Figure 5 As shown, the image processing device 500 includes a first receiving module 510, a first generating module 520, and a sending module 530.

[0130] The first receiving module 510 is used to receive the photo-taking command;

[0131] The first generation module 520 is used to generate an original image in response to the photo-taking command; wherein the original image is the image output by the image sensor;

[0132] The sending module 530 is used to send the original image to the control device so that the control device can perform image processing on the original image to generate a photographed image.

[0133] In one embodiment, the first receiving module 510 is specifically used to receive a photo-taking command input by a user for the head-mounted display device; or, to receive a photo-taking command sent by a control device to the head-mounted display device.

[0134] In one embodiment, the device 500 further includes a startup module (not shown).

[0135] The first receiving module 510 is also used to receive a startup command to start the camera application;

[0136] A startup module, used to launch the camera application in response to the startup command;

[0137] The first generation module 520 is also configured to generate a first preview image when the camera application is in the startup state.

[0138] According to embodiments of this disclosure, upon receiving a photo-taking command, the head-mounted display device generates an original image in response to the command. This original image is the image output by the image sensor. The original image is then sent to a control device, which performs image processing on the original image to generate a photographed image. In other words, by using a high-power, offline-processing method for photo-taking—sending the original image output by the image sensor at the head-mounted display device to the control device for image processing to generate a photographed image—the power consumption of the head-mounted display device is effectively reduced.

[0139] <Device Embodiment Two>

[0140] Figure 6 This is a schematic diagram of an image processing apparatus according to one embodiment, applied to a control device. (Refer to...) Figure 6 As shown, the image processing device 600 includes a second receiving module 610 and a second generating module 620.

[0141] The second receiving module 610 is used to receive the original image sent by the head-mounted display device; wherein, the original image is the image output by the image sensor in the camera device of the head-mounted display device when the head-mounted display device receives a photo-taking command;

[0142] The second generation module 620 is used to perform image processing on the original image to generate a photographed image.

[0143] In one embodiment, the second generation module 620 is used to perform image processing on the original image through the image signal processor of the control device to generate the captured image;

[0144] The image signal processor includes at least a raw image processing module, an image back-end processing module, and an image encoding module.

[0145] In one embodiment, the second generation module 620 is further configured to control the control device to enter a test image mode; when the control device enters the test mode, receive a pre-stored test image through the camera serial interface of the control device; and generate a second preview image based on the test image.

[0146] In one embodiment, the image size of the test image is the same as the image size of the original image.

[0147] According to embodiments of this disclosure, when a head-mounted display device receives a photo-taking command, it generates an original image in response to the command. This original image is the image output by the image sensor. The original image is then sent to a control device, whereby the control device processes the original image to generate a photographed image. In other words, by using a high-power, offline-processing method for photo-taking—sending the original image output by the image sensor at the head-mounted display device to the control device for image processing to generate a photographed image—the power consumption of the head-mounted display device is effectively reduced.

[0148] <Equipment Example>

[0149] Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to one embodiment. For example... Figure 7As shown, the electronic device 700 includes a processor 710 and a memory 720.

[0150] The memory 720 can be used to store executable computer instructions.

[0151] The processor 710 can be used to execute the image processing method according to the method embodiments of this disclosure, under the control of the executable computer instructions.

[0152] The electronic device 700 can be as follows: Figure 1 The head-mounted display device 100 shown can also be Figure 1 The control device 200 shown.

[0153] In another embodiment, the electronic device 700 may include the above-mentioned image processing device 500 or the above-mentioned image processing device 600.

[0154] In one embodiment, each module of the above image processing apparatus 500 or the above image processing apparatus 600 can be implemented by the processor 710 running computer instructions stored in the memory 720.

[0155] <Equipment Example>

[0156] Figure 8 This is a schematic diagram of the hardware structure of an image processing system according to one embodiment. For example... Figure 8 As shown, the image processing system 800 includes a head-mounted display device 810 and a control device 820, which are communicatively connected.

[0157] The head-mounted display device 810 is configured to receive a photo-taking command, generate an original image in response to the photo-taking command, and send the original image to the control device 820; wherein the original image is an image output by the image sensor in the camera device of the head-mounted display device 810;

[0158] The control device 820 is used to receive the original image sent by the head-mounted display device 810, and to perform image processing on the original image to generate a photographed image.

[0159] Computer-readable storage media

[0160] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the image processing method provided in this disclosure.

[0161] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0162] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0163] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0164] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0165] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0166] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0167] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0169] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. An image processing method, characterized in that, An image processing system is applied, the image processing system including a head-mounted display device and a control device, the head-mounted display device and the control device being communicatively connected, the head-mounted display device including a camera device, the camera device including an image sensor, the method comprising: The head-mounted display device receives a photo-taking command; In response to the photographing command, an original image is generated; wherein the original image is the image output by the image sensor; and The original image is sent to the control device so that the control device can perform image processing on the original image to generate a photographed image; Before receiving the original image sent by the head-mounted display device, the control device first enters a test image mode. In this mode, the control device obtains a pre-stored test image data stream through its camera serial interface decoder (CSID). After receiving the test image data stream, the CSID sends it to the image front-end processing module (IFE). The IFE processes the test image data stream and stores it in its local memory (DDR). The image back-end processing module (IPE) obtains the processed test image data stream from the local DDR, processes it, and stores it in the local DDR. The JPEG generation module obtains the processed MIPI data stream from the local DDR and processes it to generate a preview image.

2. The method according to claim 1, characterized in that, The receiving of the photo-taking command includes: Receive a photo-taking command input by the user for the head-mounted display device; or, The device receives a photo-taking command sent from the control device to the head-mounted display device.

3. The method according to claim 1, characterized in that, Before receiving the photo-taking command, the method further includes: Receive the launch command to start the camera application; In response to the launch command, the camera application is launched; With the camera application running, a first preview image is generated.

4. An image processing method, characterized in that, Applied to control equipment, the method includes: Receive raw images sent by a head-mounted display device; wherein, the raw images are images output by the image sensor in the camera device of the head-mounted display device when the head-mounted display device receives a photo-taking command; The original image is processed to generate a photographed image; Before receiving the original image sent by the head-mounted display device, the control device first enters a test image mode. In this mode, the control device obtains a pre-stored test image data stream through its camera serial interface decoder (CSID). After receiving the test image data stream, the CSID sends it to the image front-end processing module (IFE). The IFE processes the test image data stream and stores it in its local memory (DDR). The image back-end processing module (IPE) obtains the processed test image data stream from the local DDR, processes it, and stores it in the local DDR. The JPEG generation module obtains the processed MIPI data stream from the local DDR and processes it to generate a preview image.

5. The method according to claim 4, characterized in that, The step of processing the original image to generate a photographed image includes: The image signal processor of the control device performs image processing on the original image to generate the captured image; The image signal processor includes at least a raw image processing module, an image back-end processing module (IPE), and an image encoding module.

6. The method according to claim 4, characterized in that, The image size of the test image in the test image data stream is the same as the image size of the original image.

7. An image processing apparatus, characterized in that, Applied to control equipment, the device includes: The second receiving module is used to receive the original image sent by the head-mounted display device; wherein, the original image is the image output by the image sensor in the camera device of the head-mounted display device when the head-mounted display device receives a photo-taking command; The second generation module is used to perform image processing on the original image to generate a photographed image; The second generation module is further configured to control the control device to enter a test image mode before receiving the original image sent by the head-mounted display device. When the control device enters the test image mode, the control device obtains a pre-stored test image data stream through its own camera serial interface decoder CSID. After receiving the test image data stream, CSID sends the test image data stream to the image front-end processing module IFE. The image front-end processing module IFE processes the test image data stream and stores it in the local DDR. The image back-end processing module IPE obtains the processed test image data stream from the local DDR, processes the processed test image data stream, and stores it in the local DDR. The JPEG generation module obtains the processed MIPI data stream from the local DDR and processes the processed MIPI data stream to generate a preview image.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable computer instructions; A processor, configured to execute the image processing method according to any one of claims 1-6 under the control of the executable computer instructions.

9. An image processing system, characterized in that, The image processing system includes a head-mounted display device and a control device, which are communicatively connected. The head-mounted display device is configured to receive a photo-taking command, and in response to the photo-taking command, generate an original image and send the original image to the control device; wherein the original image is an image output by the image sensor in the camera device of the head-mounted display device; The control device is used to receive the original image sent by the head-mounted display device, and to perform image processing on the original image to generate a photographed image; The control device is further configured to enter a test image mode before receiving the original image sent by the head-mounted display device. When the control device enters the test image mode, it obtains a pre-stored test image data stream through its own camera serial interface decoder CSID. After receiving the test image data stream, CSID sends the test image data stream to the image front-end processing module IFE. The image front-end processing module IFE processes the test image data stream and stores it in the local DDR. The image back-end processing module IPE obtains the processed test image data stream from the local DDR, processes the processed test image data stream, and stores it in the local DDR. The JPEG generation module obtains the processed MIPI data stream from the local DDR and processes the processed MIPI data stream to generate a preview image.

Images