Image processing method and apparatus, device, and system

Abstract

Disclosed in the present invention are an image processing method and apparatus, a device, and a system, applied to a head-mounted display device. The head-mounted display device comprises a camera apparatus, and the camera apparatus comprises an image sensor. The method comprises: receiving a capturing instruction; generating an original image in response to the capturing instruction, wherein the original image is an image outputted by the image sensor; and sending the original image to a control device, so that the control device performs image processing on the original image to generate a captured image.

Description

Image processing method, device, equipment and system

[0001] The present application claims priority to the Chinese patent application No. 202411044632.8, filed on July 31, 2024, and entitled "Image processing method, device, equipment and system", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of head-mounted display devices, and more particularly, to an image processing method, an image processing device, an electronic device, and an image processing system. BACKGROUND

[0003] Power consumption is a key indicator for measuring the success of augmented reality (AR) products, which affects the overall design of the product. Good power consumption control can equip a low-capacity battery, thereby reducing the overall weight of the product. However, the camera is a high-power item in the AR product, and its current can reach 1A at the moment of shooting, which seriously increases the power consumption of the AR product. Therefore, optimizing the shooting power consumption can effectively reduce the overall power consumption of the AR product. SUMMARY

[0004] Embodiments of the present application aim to provide an image processing method, device, equipment and system.

[0005] According to a first aspect of embodiments of the present application, an image processing method is provided, applied to a head-mounted display device, the head-mounted display device comprising a camera device, the camera device comprising an image sensor, the method comprising:

[0006] receiving a shooting instruction;

[0007] generating an original image in response to the shooting instruction; wherein the original image is an image output by the image sensor;

[0008] sending the original image to a control device for the control device to generate a shooting image by image processing the original image.

[0009] Optionally, the receiving a shooting instruction comprises:

[0010] receiving a shooting instruction input by a user for the head-mounted display device; or,

[0011] receiving a shooting instruction sent by a control device to the head-mounted display device.

[0012] Optionally, before the receiving a shooting instruction, the method further comprises:

[0013] receiving a starting instruction of starting a camera application;

[0014] starting the camera application in response to the starting instruction;

[0015] generating a first preview image in a case that the camera application is in a starting state.

[0016] According to a second aspect of the embodiment of the present application, an image processing method applied to a control device is provided, and the method comprises:

[0017] receiving an original image sent by a head-mounted display device; wherein the original image is an image output by an image sensor in a camera of the head-mounted display device in a case that the head-mounted display device receives a photographing instruction;

[0018] performing image processing on the original image to generate a photographing image.

[0019] Optionally, the image processing on the original image to generate a photographing image comprises:

[0020] performing image processing on the original image by an image signal processor of the control device to generate the photographing image;

[0021] wherein the image signal processor at least comprises an original image processing module, an image back-end processing module and an image encoding module.

[0022] Optionally, before the receiving an original image sent by a head-mounted display device, the method further comprises:

[0023] controlling the control device to enter a test image mode;

[0024] acquiring a pre-stored test image by a camera serial interface decoder of the control device in a case that the control device enters the test mode;

[0025] generating a second preview image according to the test image.

[0026] According to a third aspect of the embodiment of the present application, an image processing device applied to a head-mounted display device is provided, the head-mounted display device comprises a camera, and the camera comprises an image sensor, and the device comprises:

[0027] a first receiving module configured to receive a photographing instruction;

[0028] a first generating module configured to generate an original image in response to the photographing instruction; wherein the original image is an image output by the image sensor;

[0029] The sending module is configured to send the original image to the control device, so that the control device generates a photograph image by performing image processing on the original image.

[0030] According to a fourth aspect of the embodiments of the present application, an image processing device is provided, which is applied to a control device, and the device comprises:

[0031] The second receiving module is configured to receive an original image sent by the head-mounted display device, wherein the original image is an image output by an image sensor in a camera of the head-mounted display device when the head-mounted display device receives a photograph instruction.

[0032] The second generating module is configured to generate a photograph image by performing image processing on the original image.

[0033] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, which comprises:

[0034] The memory is configured to store executable computer instructions.

[0035] The processor is configured to execute the image processing method according to the first aspect above under the control of the executable computer instructions.

[0036] According to a sixth aspect of the embodiments of the present application, an image processing system is provided, which comprises a head-mounted display device and a control device, and the head-mounted display device and the control device are communicatively connected,

[0037] The head-mounted display device is configured to receive a photograph instruction, generate an original image in response to the photograph instruction, and send the original image to the control device, wherein the original image is an image output by an image sensor in a camera of the head-mounted display device.

[0038] The control device is configured to receive the original image sent by the head-mounted display device, and generate a photograph image by performing image processing on the original image.

[0039] An advantage of the embodiments of the present application is that the head-mounted display device generates an original image in response to a photograph instruction when the head-mounted display device receives the photograph instruction, wherein the original image is an image output by an image sensor, and the head-mounted display device sends the original image to the control device, and the control device generates a photograph image by performing image processing on the original image. That is, the photograph process of the head-mounted display device, which is high in power consumption and can be processed offline, is improved, the original image output by the image sensor of the head-mounted display device is sent to the control device for image processing to generate a photograph image, and the power consumption of the head-mounted display device is effectively reduced.

[0040] Other features and advantages of the present specification will become apparent from the following detailed description of illustrative embodiments thereof, taken in conjunction with the accompanying drawings.

[0041] Other features and advantages of the present specification will become apparent from the following detailed description of illustrative embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments of the present specification and, together with the description, serve to explain the principles of the present specification.

[0043] FIG. 1 is a schematic diagram of a hardware configuration of an image processing system according to an embodiment of the present specification;

[0044] FIG. 2 is a schematic diagram of a flow of an image processing method according to an embodiment of the present specification;

[0045] FIG. 3 is a schematic diagram of a flow of an image processing method according to another embodiment of the present specification;

[0046] FIG. 4 is a schematic diagram of an interaction flow between a head-mounted display device and a control device according to an example;

[0047] FIG. 5 is a schematic diagram of a principle block of an image processing apparatus according to an embodiment of the present specification;

[0048] FIG. 6 is a schematic diagram of a principle block of an image processing apparatus according to another embodiment of the present specification;

[0049] FIG. 7 is a schematic diagram of a principle block of an electronic device according to an embodiment of the present specification;

[0050] FIG. 8 is a schematic diagram of a principle block of an image processing system according to an embodiment of the present specification.

[0051] The implementation, functional features and advantages of the present specification will be further described with reference to the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present specification will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, rather than all the embodiments of the present specification. Based on the embodiments in the present specification, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present specification.

[0053] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications will also change accordingly.

[0054] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. For example, "A and / or B" includes A scheme, or B scheme, or A and B simultaneously satisfy the scheme. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope required by the present application.

[0055] <Hardware configuration>

[0056] Fig. 1 is a schematic diagram of a hardware configuration of an image processing system that can be used to implement an embodiment. Fig. 1 shows a head-mounted display device 100 and a control device 200. The head-mounted display device 100 and the control device 100 are communicatively connected.

[0057] In an embodiment, as shown in Fig. 1, the head-mounted display device 100 can at least include a processor 101, a memory 102, a communication device 103, a camera device 104. In addition, the head-mounted display device 100 can further include a power management chip 105 and a battery 106, etc. It should be noted that the head-mounted display device 100 can have a display device (not shown in the figure), and can also not have a display device, wherein the display device can be a display screen.

[0058] The processor 101 can be various processors. The memory 102 can store underlying software, system software, application software, data, etc. required for the head-mounted display device 100 to run. The memory 102 can include various forms of memory, such as ROM, RAM, Flash, etc., wherein the RAM can be a Double Data Rate SDRAM (Double Data Rate SDRAM), abbreviated as DDR. The communication device 103 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G and 5G communication device, etc. The camera 104 can be a single camera or a multi-camera. The power management chip 105 is used to manage the input power of the head-mounted display device 100, and can also manage the battery 106 to ensure high utilization efficiency. The battery 106 is a lithium ion battery, etc.

[0059] The head-mounted display device 100 can be AR (Augmented Reality) glasses, MR (Mixed Reality) glasses, etc., and the embodiments of the present application are not limited thereto. The components shown in FIG. 1 are merely illustrative. The head-mounted display device 100 can include one or more of the components shown in FIG. 1, and does not necessarily include all the components in FIG. 1. The head-mounted display device 100 shown in FIG. 1 is merely explanatory, and is by no means intended to limit the embodiments, applications or uses thereof.

[0060] In the present embodiment, the memory 102 of the head-mounted display device 100 is used to store program instructions for controlling the processor 101 to operate to perform the image processing method, and the skilled person can design the instructions according to the disclosed scheme. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0061] In one embodiment, as shown in FIG. 1, the control device 200 can include a processor 201, a memory 202, a communication device 203, a display device 204, a camera 205, etc. In addition, the control device 200 can also include a power management chip 206 and a battery 207, etc.

[0062] The processor 201 can be various processors. The memory 202 can store underlying software, system software, application software, data, etc. required for the operation of the device 200. The memory 202 can include various forms of memory, such as ROM, RAM, Flash, etc., wherein the RAM can be DDR. The communication device 203 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G and 5G communication device, etc. The display device 204 can be a touch screen. The camera device 205 can be a single camera or a multi-camera. The power management chip 206 is used to manage the input power of the control device 200, and can also manage the battery 207 to ensure high utilization efficiency. The battery 207 is a lithium ion battery or the like.

[0063] The control device 200 can be an Android device, such as a mobile phone, a tablet computer, a palm computer, a PUCK, etc. The components shown in FIG. 1 are merely illustrative. The control device 200 can include one or more of the components shown in FIG. 1, and does not necessarily include all the components in FIG. 1. The control device 200 shown in FIG. 1 is merely illustrative, and is by no means intended to limit the embodiments, applications or uses herein.

[0064] Taking the PUCK as an example, the PUCK includes the processor 201, the memory 202, the communication device 203 and the camera device 205, and the camera device 205 includes an image signal processor (ISP) including a raw image processing module (Bayer Process Segment, BPS), an image front-end processing module (IFE), an image processing engine (IPE) and an image encoding module. The PUCK can be used to control a head-mounted display device, such as AR glasses, and can also be used to receive a raw image sent by the head-mounted display device and process the raw image through the image signal processor ISP to obtain a photographed image.

[0065] In this embodiment, the memory 202 of the control device 200 is used to store program instructions for controlling the processor 201 to operate to perform an image processing method. The skilled person can design the instructions according to the disclosed scheme. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0066] It should be understood that although FIG. 1 only shows one head-mounted display device 100 and one control device 200, this does not mean that the number of each is limited, and the image processing system can include multiple head-mounted display devices 100 and multiple control devices 200.

[0067] In the foregoing description, the skilled person can design the instructions according to the scheme provided by the present application. How the instructions control the processor to operate is known in the art, and thus is not described in detail herein.

[0068] <Method Embodiment One>

[0069] FIG. 2 shows an image processing method of an embodiment of the present application, which can be implemented by the head-mounted display device shown in FIG. 1, the head-mounted display device comprising a camera device, the camera device comprising an image sensor Sensor. As shown in FIG. 2, the image processing method of this embodiment can comprise the following steps S2100-S2300:

[0070] Step S2100, receiving a photographing instruction.

[0071] Optionally, the step S2100 of receiving a photographing instruction can further comprise: receiving a photographing instruction input by a user to the head-mounted display device.

[0072] Specifically, in the case that the camera application of the head-mounted display device is in an open state, the user can input a photographing instruction through a photographing control provided by the camera application of the head-mounted display device. Exemplarily, the photographing instruction can be a touch input of the user to the photographing control of the camera application, and the touch input can be a click input.

[0073] Optionally, the step S2100 of receiving a photographing instruction can further comprise: receiving a photographing instruction sent by a control device to the head-mounted display device.

[0074] It should be noted that, on the one hand, the head-mounted display device and the control device are in communication connection, which can be a WiFi connection. On the other hand, the control device usually has a control application installed therein, and the control device can control the head-mounted display device based on the control application.

[0075] Specifically, in the case that the control application in the control device is in a running state, the control device can send a photographing instruction to the head-mounted display device through the control application based on the communication connection between the head-mounted display device and the control device.

[0076] In an alternative embodiment, before performing the step S2100 of receiving a photographing instruction, the image processing method of the embodiment of the present application further comprises a step S4100 of: receiving a starting instruction for starting a camera application; in response to the starting instruction, starting the camera application; and generating a first preview image in the case that the camera application is in a starting state.

[0077] The photographing device further comprises an image signal processor (ISP). The image signal processor (ISP) comprises at least an image front-end processing module (IFE), an image post-end processing module (IPE) and an image encoding module.

[0078] The image front-end processing module (IFE) is configured to perform color correction processing, down-sampling processing, demosaicing statistical 3A data processing and the like on the image. It should be noted that the image processed by the image front-end processing module (IFE) is in YUV format, wherein Y represents brightness, and U and V represent chrominance.

[0079] The image post-end processing module (IPE) is configured to perform noise reduction processing, cropping processing, color processing, detail enhancement processing and the like on the image.

[0080] The image encoding module is configured to perform format conversion on the image. The image encoding module can be a JPEG (Joint Photographic Experts Group) generation module, that is, the image processed by the image encoding module is in JPEG format.

[0081] Specifically, the head-mounted display device is provided with a camera application. A user can input a start instruction through an icon of the camera application provided by the head-mounted display device to start the camera application. Exemplarily, the start instruction can be a touch input of the user on the icon of the camera application, and the touch input can be a click input.

[0082] Referring to FIG. 4, for the head-mounted display device, when the camera application of the head-mounted display device is in a started state, the image sensor Sensor outputs a MIPI (Mobile Industry Processor Interface) data stream, and the MIPI data stream is received through a camera serial interface decoder (CSID). After receiving the MIPI data stream, the CSID can send the MIPI data stream to the image front-end processing module (IFE). The image front-end processing module (IFE) stores the processed MIPI data stream to the local DDR after processing the MIPI data stream. The image post-end processing module (IPE) obtains the processed MIPI data stream from the local DDR, processes the processed MIPI data stream, and stores the processed MIPI data stream to the local DDR. The JPEG generation module obtains the processed MIPI data stream from the local DDR, processes the processed MIPI data stream (not shown in FIG. 4), and generates a preview image Preview.

[0083] In a specific implementation, the head-mounted display device can first execute step S4100 to control the camera application to be in an opened state, and generate a preview image in real time, and then execute step S2100 to receive a photographing instruction.

[0084] After the above step S2100 of receiving the photographing instruction is performed, proceed to:

[0085] Step S2200, in response to the photographing instruction, generate a raw image.

[0086] The raw image is an image output by the image sensor. Generally, the raw image can be referred to as a RAW image.

[0087] In this embodiment, after the head-mounted display device receives the photographing instruction, the head-mounted display device can generate a raw image through the image sensor in response to the photographing instruction.

[0088] Referring to FIG. 4, when the head-mounted display device receives the photographing instruction, the head-mounted display device generates a RAW image through the image sensor Sensor and stores the RAW image to the local DDR.

[0089] After the above step S2200 of generating a raw image in response to the photographing instruction is performed, proceed to:

[0090] Step S2300, send the raw image to the control device, so that the control device generates a photographing image through image processing on the raw image.

[0091] In this embodiment, after the image sensor generates a raw image, the head-mounted display device first stores the raw image to the local DDR, and then sends the raw image to the control device through a communication connection such as a WiFi connection established between the head-mounted display device and the control device, so that the control device generates a photographing image through image processing on the raw image.

[0092] In a specific implementation, after the control device receives the raw image, the control device generates a photographing image through image processing on the raw image by an image signal processor ISP of the control device.

[0093] The image signal processor ISP includes a raw image processing module BPS, an image back-end processing module IPE, and an image encoding module.

[0094] The raw image processing module BPS is referred to as a RAW image processing module, and is configured to perform bad pixel removal processing, phase focusing processing, demosaicing processing, downsampling processing, HDR processing, and Bayer mixed noise reduction processing on the raw image. It should be noted that the image processed by the raw image processing module is in YUV format.

[0095] The image back-end processing module IPE is configured to perform noise reduction processing, cropping processing, color processing, detail enhancement processing, and the like on the image.

[0096] The image encoding module is used for format conversion of the image. The image encoding module can be a JPEG generation module, that is, the format of the image processed by the image encoding module is JPEG format.

[0097] Referring to FIG. 4, the control device receives the RAW image sent by the head-mounted display device and stores the RAW image to the local DDR. The raw image processing module BPS obtains the RAW image from the local DDR and stores the processed RAW image to the local DDR after processing the RAW image. The image post-processing module IPE obtains the processed RAW image from the local DDR and stores the processed RAW image to the local DDR after processing the processed RAW image. The JPEG generation module obtains the processed RAW image from the local DDR and generates a photographed image JPEG after processing the processed RAW image (not shown in the figure).

[0098] It should be noted that, since the camera application of the control device is in a closed state, before the control device receives the RAW image sent by the head-mounted display device, the control device can enter a test image mode first, and in the case of entering the test image mode, the control device obtains a pre-stored test image through the camera serial interface decoder CSID of the control device, and generates a second preview image according to the test image. The image size of the test image is the same as the image size of the RAW image.

[0099] In a popular way, the control device generates a test image as its preview image (which can not be displayed), and waits for a photographing event at this time. Receiving the RAW image sent by the head-mounted display device is equivalent to triggering a photographing event, and the received RAW image can be processed.

[0100] The image signal processor ISP of the control device further includes an image post-processing module IPE.

[0101] Referring to FIG. 4, before the control device receives the RAW image sent by the head-mounted display device, the control device can control itself to enter a test image mode first. In the case of entering the test image mode, the control device obtains a test image data stream through the camera serial interface decoder CSID of the control device. After receiving the test image data stream, the CSID can send the test image data stream to the image front-end processing module IFE. The image front-end processing module IFE stores the processed test image data stream to the local DDR after processing the test image data stream. The image post-processing module IPE obtains the processed test image data stream from the local DDR and stores the processed test image data stream to the local DDR after processing the processed test image data stream. The JPEG generation module obtains the processed MIPI data stream from the local DDR and generates a preview image after processing the processed MIPI data stream (not shown in the figure). It should be noted that the control device generates a preview image based on the test image, and usually discards the generated preview image.

[0102] According to the embodiment of the present application, the head-mounted display device generates the original image in response to the photographing instruction in the case that the photographing instruction is received, wherein the original image is the image output by the image sensor, and sends the original image to the control device, and the control device can generate the photographing image by image processing the original image. That is, for the photographing process of the head-mounted display device which is high in power consumption and can be processed offline, the original image output by the image sensor of the head-mounted display device is sent to the control device for image processing to generate the photographing image, thereby effectively reducing the power consumption of the head-mounted display device.

[0103] <Method Embodiment Two>

[0104] FIG. 3 shows an image processing method according to an embodiment of the present application, which can be implemented by the control device shown in FIG. 1. As shown in FIG. 3, the image processing method according to the embodiment can include the following steps S3100-S3200:

[0105] Step S3100, receiving the original image sent by the head-mounted display device.

[0106] The original image is the image output by the image sensor in the camera of the head-mounted display device in the case that the head-mounted display device receives the photographing instruction.

[0107] After the above step S3100 of receiving the original image sent by the head-mounted display device, the following step is performed:

[0108] Step S3200, image processing the original image to generate the photographing image.

[0109] In an optional embodiment, the step S3200 of image processing the original image to generate the photographing image can further include: image processing the original image by the image signal processor of the control device to generate the photographing image.

[0110] The image signal processor at least includes an original image processing module BPS, an image post-processing module IPE and an image encoding module.

[0111] Referring to FIG. 4, the control device receives the RAW image sent by the head-mounted display device and stores the RAW image to the local DDR. The original image processing module BPS acquires the RAW image from the local DDR, processes the RAW image, and stores the processed RAW image to the local DDR. The image post-processing module IPE acquires the processed RAW image from the local DDR, processes the processed RAW image, and stores the processed RAW image to the local DDR. The JPEG generation module acquires the processed RAW image from the local DDR, processes the processed RAW image (not shown in the figure), and generates a photographed image JPEG.

[0112] In an optional embodiment, before performing the above step S3100 of receiving the RAW image sent by the head-mounted display device, the image processing method of the embodiment of the present application further comprises: controlling the control device to enter a test image mode; in the case where the control device enters the test mode, acquiring a pre-stored test image through a camera serial interface of the control device; and generating a second preview image according to the test image.

[0113] The image size of the test image is the same as the image size of the RAW image.

[0114] It should be noted that, since the camera application of the control device is in a closed state, before receiving the RAW image sent by the head-mounted display device, the control device can first enter the test image mode, acquire a pre-stored test image through a camera serial interface decoder CSID of the control device in the case where the control device enters the test image mode, and generate a second preview image according to the test image.

[0115] In a popular way, the control device generates the test image as its preview image (which can not be displayed), waits for a photographing event, and processes the received RAW image after receiving the RAW image sent by the head-mounted display device, which is equivalent to triggering a photographing event.

[0116] The image signal processor ISP of the control device further comprises an image post-processing module IFE.

[0117] Referring to FIG. 4, before receiving the RAW image sent by the head-mounted display device, the control device can control itself to enter a test image mode first. In the case where the control device enters the test image mode, the control device acquires a test image data stream through a camera serial interface decoder (CSID) of the control device. After receiving the test image data stream, the CSID sends the test image data stream to an image front-end processing module (IFE). The IFE stores the processed test image data stream to a local DDR after processing the test image data stream. An image post-processing module (IPE) acquires the processed test image data stream from the local DDR and stores the processed test image data stream to the local DDR after processing the processed test image data stream. A JPEG generation module acquires the processed MIPI data stream from the local DDR and generates a preview image after processing the processed MIPI data stream (not shown in the figure). It should be noted that the control device generates a preview image based on the test image, and usually discards the generated preview image.

[0118] According to the embodiment of the present application, since the head-mounted display device generates an original image in response to the photographing instruction in the case where the photographing instruction is received, and sends the original image to the control device, the control device can generate a photographing image by processing the original image. That is, the high-power and offline processing photographing is effectively reduced by sending the original image output by the image sensor of the head-mounted display device to the control device for image processing to generate a photographing image.

[0119] <Example>

[0120] Next, referring to FIG. 4, taking the head-mounted display device as AR glasses and the control device as PUCK as an example, an image processing method is shown in an example, and the image processing method can further include:

[0121] In step S401, after the AR glasses start the camera application, the image sensor Sensor outputs a MIPI data stream.

[0122] In step S402, the AR glasses receive the MIPI data stream through the CSID and send the MIPI data stream to the IFE module. The IFE module stores the processed MIPI data stream to a local DDR after processing the MIPI data stream. The IPE module acquires the processed MIPI data stream from the local DDR and stores the acquired processed MIPI data stream to the local DDR after processing the acquired processed MIPI data stream. The JPEG generation module acquires the processed MIPI data stream from the local DDR and generates a first preview image Preview after processing the processed MIPI data stream.

[0123] In step S403, the AR glasses receive a photographing instruction.

[0124] At step S404, the AR glasses output a RAW image through the image sensor Sensor in response to the photographing instruction, and store the RAW image to the local DDR.

[0125] At step S405, the AR glasses acquire the RAW image from the local DDR, and send the RAW image to the PUCK through the WiFi connection between the AR glasses and the PUCK.

[0126] At step S501, the PUCK acquires a test image data stream through the CSID before receiving the RAW image, and sends the test image data stream to the IFE module. The IFE module stores the processed test image data stream to the local DDR. The IPE module acquires the processed test image data stream from the local DDR, processes the acquired test image data stream, and stores the processed test image data stream to the local DDR. The JPEG generation module acquires the processed test image data stream from the local DDR, processes the acquired processed test image data stream, and generates a second preview image (discarded).

[0127] At step S502, the PUCK receives the RAW image sent by the AR glasses and stores the RAW image to the local DDR. The BPS module acquires the RAW image from the local DDR, processes the acquired RAW image, and stores the processed RAW image to the local DDR. The IPE module acquires the processed RAW image from the local DDR, processes the acquired RAW image, and stores the processed RAW image to the local DDR. The JPEG generation module acquires the processed RAW image from the local DDR, and generates a photographing image based on the acquired processed RAW image.

[0128] According to the present example, by sending the RAW image of the AR glasses to the image signal processor of the PUCK to process and encode the photographing, the power consumption of the AR glasses is effectively reduced.

[0129] <Device Embodiment One>

[0130] FIG. 5 is a schematic diagram of an image processing device according to an embodiment, applied to a head-mounted display device, the head-mounted display device comprising a camera device, the camera device comprising an image sensor. Referring to FIG. 5, the image processing device 500 comprises a first receiving module 510, a first generating module 520, and a sending module 530.

[0131] The first receiving module 510 is configured to receive a photographing instruction.

[0132] The first generating module 520 is configured to generate a RAW image in response to the photographing instruction, wherein the RAW image is an image output by the image sensor.

[0133] The sending module 530 is configured to send the original image to the control device, so that the control device generates a photographed image by performing image processing on the original image.

[0134] In one embodiment, the first receiving module 510 is specifically configured to receive a photographed instruction input by a user for the head-mounted display device, or receive a photographed instruction sent by the control device to the head-mounted display device.

[0135] In one embodiment, the apparatus 500 further includes a starting module (not shown in the figure).

[0136] The first receiving module 510 is further configured to receive a starting instruction for starting a camera application.

[0137] The starting module is configured to start the camera application in response to the starting instruction.

[0138] The first generating module 520 is further configured to generate a first preview image in a case where the camera application is in a starting state.

[0139] According to the embodiment of the present application, the head-mounted display device generates an original image in response to a photographed instruction in a case where the photographed instruction is received, wherein the original image is an image output by an image sensor, and the head-mounted display device sends the original image to the control device, and the control device generates a photographed image by performing image processing on the original image. That is, the head-mounted display device effectively reduces the power consumption by sending the original image output by the image sensor to the control device to generate a photographed image by performing image processing on the original image.

[0140] <Embodiment two of the apparatus>

[0141] FIG. 6 is a schematic diagram of an image processing apparatus according to one embodiment, which is applied to a control device. Referring to FIG. 6, the image processing apparatus 600 includes a second receiving module 610 and a second generating module 620.

[0142] The second receiving module 610 is configured to receive an original image sent by a head-mounted display device, wherein the original image is an image output by an image sensor in a camera of the head-mounted display device in a case where the head-mounted display device receives a photographed instruction.

[0143] The second generating module 620 is configured to perform image processing on the original image to generate a photographed image.

[0144] In one embodiment, the second generating module 620 is configured to perform image processing on the original image by an image signal processor of the control device to generate the photographed image.

[0145] The image signal processor comprises at least an original image processing module, an image back-end processing module and an image encoding module.

[0146] In one embodiment, the second generation module 620 is further configured to control the control device to enter a test image mode, and receive a pre-stored test image through a camera serial interface of the control device when the control device enters the test mode, and generate a second preview image according to the test image.

[0147] In one embodiment, the test image has the same image size as the original image.

[0148] According to the embodiment of the present application, when the head-mounted display device receives a photographing instruction, the head-mounted display device generates an original image in response to the photographing instruction, where the original image is an image output by an image sensor, and sends the original image to the control device, so that the control device generates a photographing image by processing the original image. That is, the head-mounted display device effectively reduces the power consumption by sending the original image output by the image sensor to the control device to generate a photographing image.

[0149] <Device Embodiment>

[0150] FIG. 7 is a schematic diagram of a hardware structure of an electronic device according to one embodiment. As shown in FIG. 7, the electronic device 700 comprises a processor 710 and a memory 720.

[0151] The memory 720 can be used to store executable computer instructions.

[0152] The processor 710 can be used to execute the image processing method according to the embodiment of the method of the present application under the control of the executable computer instructions.

[0153] The electronic device 700 can be the head-mounted display device 100 shown in FIG. 1, or the control device 200 shown in FIG. 1.

[0154] In another embodiment, the electronic device 700 can comprise the image processing apparatus 500 or the image processing apparatus 600.

[0155] In one embodiment, each module of the image processing apparatus 500 or the image processing apparatus 600 can be realized by running computer instructions stored in the memory 720 by the processor 710.

[0156] <Device Embodiment>

[0157] Figure 8 is a schematic diagram of the hardware structure of an image processing system according to one embodiment. As shown in Figure 8, the image processing system 800 includes a head-mounted display device 810 and a control device 820, which are communicatively connected.

[0158] The head-mounted display device 810 is configured to receive a photo-taking command, generate an original image in response to the photo-taking command, and send the original image to the control device 820; wherein the original image is an image output by the image sensor in the camera device of the head-mounted display device 810;

[0159] The control device 820 is used to receive the original image sent by the head-mounted display device 810, and to perform image processing on the original image to generate a photographed image.

[0160] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0161] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0162] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0163] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0164] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0165] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0166] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0169] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, television, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0171] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. An image processing method, characterized by, The method is applied to a head-mounted display device, the head-mounted display device comprises a camera device, the camera device comprises an image sensor, and the method comprises the following steps: receiving a photographing instruction; generating an original image in response to the photographing instruction; wherein the original image is an image output by the image sensor; sending the original image to a control device for the control device to generate a photographing image by image processing on the original image.

2. The method of claim 1, wherein, The receiving of the photographing instruction comprises: receiving a photographing instruction input by a user for the head-mounted display device; or receiving a photographing instruction sent by the control device to the head-mounted display device.

3. The method of claim 1, wherein, Before the receiving of the photographing instruction, the method further comprises: receiving a starting instruction for starting a camera application; starting the camera application in response to the starting instruction; generating a first preview image in a case where the camera application is in a starting state.

4. An image processing method characterized by, The method is applied to a control device, and the method comprises the following steps: receiving an original image sent by a head-mounted display device; wherein the original image is an image output by an image sensor in a camera device of the head-mounted display device in a case where the head-mounted display device receives a photographing instruction; generating a photographing image by image processing on the original image.

5. The method of claim 4, wherein, The generating of the photographing image by image processing on the original image comprises: generating the photographing image by image processing on the original image by an image signal processor of the control device; wherein the image signal processor at least comprises an original image processing module, an image back-end processing module and an image encoding module.

6. The method of claim 4, wherein, Before the receiving of the original image sent by the head-mounted display device, the method further comprises: controlling the control device to enter a test image mode; acquiring a pre-stored test image by a camera serial interface decoder of the control device in a case where the control device enters the test mode; generating a second preview image according to the test image.

7. The method of claim 6, wherein, The image size of the test image is the same as the image size of the original image.

8. An image processing apparatus characterized by comprising: The device is applied to a head-mounted display device, the head-mounted display device comprises a camera device, the camera device comprises an image sensor, and the device comprises the following steps: a first receiving module, configured to receive a photographing instruction; a first generating module, configured to generate an original image in response to the photographing instruction; wherein the original image is an image output by the image sensor; a sending module, configured to send the original image to a control device for the control device to generate a photographing image by image processing on the original image.

9. An image processing apparatus characterized by comprising: The device is applied to a control device, and the device comprises the following steps: a second receiving module, configured to receive an original image sent by a head-mounted display device; wherein the original image is an image output by an image sensor in a camera device of the head-mounted display device in a case where the head-mounted display device receives a photographing instruction; a second generating module, configured to generate a photographing image by image processing on the original image.

10. An electronic device, comprising: The electronic device comprises: a memory, configured to store executable computer instructions; a processor, configured to execute an image processing method according to any one of claims 1-7 under control of the executable computer instructions.

11. An image processing system, characterized by The image processing system comprises a head-mounted display device and a control device, the head-mounted display device and the control device are communicatively connected, The head-mounted display device is configured to receive a photographing instruction, generate an original image in response to the photographing instruction, and send the original image to the control device; wherein the original image is an image output by an image sensor in a camera of the head-mounted display device; The control device is configured to receive the original image sent by the head-mounted display device, and perform image processing on the original image to generate a photographing image.

Images