Image processing method, image processing apparatus, and projection device

Abstract

Provided in the present application are an image processing method, an image processing apparatus, and a projection device. The image processing method comprises: acquiring a first image, wherein the first image is captured towards a projection medium after an infrared fill light is turned on, and a projected image is displayed on the projection medium; on the basis of the first image, determining whether the projected image is blocked by a target object; and when it is determined that the projected image is blocked by the target object, determining a first region of the projected image that is blocked, and blocking a second region of a source image, wherein the second region is a region of the source image corresponding to the first region of the projected image. A user usually uses a projection device for viewing in a dimly lit environment, and therefore turning on an infrared fill light can improve the imaging quality of a CMOS module for a target object, and an intelligent eye protection function can thus be achieved on the basis of a CMOS assembly. Compared with a projection device equipped only with a CMOS module, the present application involves slightly increased cost.

Description

Image processing methods, image processing devices and projection equipment Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method, an image processing device, and a projection device. Background Technology

[0002] Currently, most projectors on the market are equipped with Complementary Metal-Oxide Semiconductor (CMOS) modules. Based on CMOS modules, projectors can achieve various intelligent sensing functions, such as: 1) autofocus; 2) automatic keystone correction; 3) automatic obstacle avoidance; and 4) automatic screen entry. In addition, some projectors are also equipped with ambient light sensors (ALS), enabling ambient light adaptation and other functions.

[0003] For projectors equipped with a Time of Flight (TOF) module, intelligent eye protection features can be further implemented. This feature uses the TOF module to detect whether a user is in the projection area and automatically turns off the projection or blocks the image in the user's area when the user is present, thereby reducing harm to the user's eyes.

[0004] However, the cost of TOF modules used to implement intelligent eye protection functions in projectors is relatively high. For example, a 10,000-dot TOF (120*90) module costs approximately 40 to 60 yuan, and even a small-area TOF (8*8) module costs 20 to 30 yuan. Therefore, TOF modules are typically only found in high-end projectors, while low-end projectors often do not have them. Because visible light CMOS modules have poor image quality in low-light environments, they struggle to recognize the human body in real time using algorithms. This means that low-end projectors equipped only with CMOS modules cannot implement intelligent eye protection functions.

[0005] Therefore, how to implement intelligent eye protection functions in low-end projectors without affecting costs or with only a slight increase in costs has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, this application provides an image processing method, an image processing apparatus, and a projection device. This projection device can achieve intelligent eye protection functionality based on its integrated CMOS components (including a CMOS module and an infrared fill light); compared to a projection device that only integrates a CMOS module, it only increases cost slightly.

[0007] In a first aspect, this application provides an image processing apparatus, which includes a processing module and a complementary metal-oxide-semiconductor (CMOS) component. The CMOS component includes a CMOS module and an infrared fill light, and the filter structure in the CMOS module allows infrared light to pass through.

[0008] The CMOS module is used to capture images, including a first image captured toward a projection medium after the infrared fill light is turned on, and the projection medium displays a projected image.

[0009] The processing module is used to determine whether the projected image is occluded by the target object based on the first image; when it is determined that the projected image is occluded by the target object, it determines the first region of the projected image that is occluded and the second region in the source image that is occluded, wherein the second region is the region in the source image corresponding to the first region in the projected image.

[0010] The image processing apparatus provided in this application may be part of a projection device or may be a projection device itself.

[0011] Since users typically watch movies using projection devices in dimly lit environments, activating the infrared fill light improves the imaging quality of the CMOS module for the target object. Furthermore, based on the image captured by the CMOS module (i.e., the first image), it can accurately determine whether the target object has entered the projection area. Once the target object is confirmed to be in the projection area, eye protection processing is performed (i.e., identifying the first area of ​​the projected image that is blocked and the second area in the source image that is blocked). Thus, when the projection device projects the source image blocking the second area onto the projection medium, the second area in the source image will be projected onto the target object. Because the second area in the source image is blocked, the harm to the user's eyes from the light emitted by the projection device can be reduced. In other words, this application, based on a CMOS component equipped with an infrared fill light, can achieve intelligent eye protection functionality. Compared to achieving intelligent eye protection functionality based on TOF, this application is less expensive.

[0012] Furthermore, this application can achieve intelligent eye protection function by adding an infrared fill light (and in some cases, a filter structure) to existing projection devices that only have a CMOS module, with only a small increase in cost.

[0013] It should be understood that projection devices (such as projectors) typically project the source image towards the projection medium, and correspondingly, the CMOS module also faces the projection medium to capture images. Therefore, when the infrared fill light is turned on, the CMOS module faces the projection medium to capture images, capturing not only the projection medium itself but also the projected image displayed on it. When a target object enters the projection area of ​​the projection device, the CMOS module, with the infrared fill light turned on, faces the projection medium to capture images, also capturing the target object.

[0014] For example, the target object may include people, animals, etc.

[0015] For example, "source image" can be understood as the image to be projected by the projection device. "Projected image" can be understood as the image displayed on the projection medium after the source image is projected onto the projection medium by the projection device; the projected image can also be called the projected screen. Here, the projection medium can refer to the entity used to carry and display the projected image, and the projection medium includes, but is not limited to, screens, projection screens, special coating materials, water surfaces, and frosted surfaces.

[0016] For example, the projection area can refer to the spatial range in which the projection device can project the source image, including the physical space between the projection device and the projection medium.

[0017] For example, the region in the source image corresponding to the first region in the projected image can be determined based on the first region and the second mapping relationship; wherein, the second mapping relationship can be determined based on the pose of the projection device, the distance between the projection device and the projection medium (i.e., the projection distance), and the projection parameters of the projection device.

[0018] For example, occluding a second region in the source image can refer to adding a dark mask, such as a black mask, to the second region in the source image.

[0019] According to the first aspect, the CMOS module includes a camera, which includes a lens, a CMOS sensor, and one or more filter structures.

[0020] For example, at least one of the one or more filter structures allows infrared light to pass through.

[0021] For example, if the camera includes a filter structure, the camera can be called a fixed filter structure camera; wherein the filter structure is an all-pass filter structure.

[0022] For example, if the camera includes multiple filter structures, the camera may be referred to as an adjustable (or switchable) filter structure camera; wherein the multiple filter structures include two or more of an all-pass filter structure, a narrowband filter structure, or an infrared cutoff filter structure.

[0023] It should be understood that in some cases, a camera may include multiple lenses, or a camera may include multiple CMOS sensors.

[0024] According to the first aspect, or any implementation of the first aspect above, the CMOS module includes multiple cameras, each of the multiple cameras including a lens, a CMOS sensor and a filter structure, and the filter structures in any two cameras are different.

[0025] In other words, the CMOS module includes multiple cameras with fixed filter structures.

[0026] According to the first aspect, or any implementation of the first aspect above, the CMOS module includes a first camera and a second camera. The first camera includes a lens, a CMOS sensor, and multiple filter structures. The second camera includes a lens, a CMOS sensor, and a filter structure.

[0027] For example, there can be one or more first cameras, and one or more second cameras.

[0028] For example, the first camera includes a lens, a CMOS sensor, and an infrared cutoff filter structure; the second camera includes a lens, a CMOS sensor, an all-pass filter structure, and a narrowband filter structure.

[0029] According to the first aspect, or any implementation of the first aspect above, the filter structure includes an all-pass filter structure or a narrowband filter structure.

[0030] For example, the filter structure may be a filter coating or a filter.

[0031] For example, the filter structure may also include a visible light filter (also known as an infrared-cut filter, IR-CUT).

[0032] According to the first aspect, or any implementation of the first aspect above, the processing module is used for:

[0033] Based on the first image, determine the region description information of the area where the target object is located in the first image; obtain the region description information of the area where the projected image is located in the first image; based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, determine whether the projected image is occluded by the target object.

[0034] In one possible approach, the region where the target object is located can refer to the area covered by the pixels contained in the target object. In this case, the descriptive region information of the region where the target object is located can refer to the coordinates of the edge pixels of the area covered by the pixels contained in the target object.

[0035] In one possible approach, the region where the target object is located can refer to the region bounded by the outer bounding box of the area covered by the pixels contained in the target object. In this case, the descriptive region information of the region where the target object is located can refer to the descriptive information of the outer bounding box (such as the center coordinates of the outer bounding box (or the coordinates of the upper left corner of the outer bounding box), the height of the outer bounding box, and the width of the outer bounding box).

[0036] In one possible approach, the region description information of the area where the projected image is located in the first image can be identified based on the first image. The projected image is typically rectangular, and the region description information of the area where the projected image is located in the first image can include the center coordinates (or the coordinates of the top-left corner of the rectangle), the height of the rectangle, and the width of the rectangle.

[0037] In one possible approach, the region description information of the projected image area in the CMOS-captured image can be read from the storage module during the process of other previously implemented sensing functions (such as autofocus, keystone correction, automatic obstacle avoidance, and automatic screen entry).

[0038] In one possible approach, the region description information of the area where the projected image was located in the previously determined first image can be read from the storage module.

[0039] For example, based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, it can be determined whether the area where the target object is located in the first image overlaps with the area where the projected image is located in the first image. If they do not overlap, it is determined that the projected image is not occluded by the target object; if they overlap, it is determined that the projected image is occluded by the target object.

[0040] According to the first aspect, or any implementation of the first aspect above, the processing module is also used for:

[0041] The method determines the difference between the region description information of the target object's location in the second image and the region description information of the target object's location in the third image. The second image is the first image captured in this current capture, and the third image is the first image captured in a previous capture. When the difference information meets a first preset condition, based on the region description information of the target object's location in the second image and the region description information of the projected image's location in the second image, it is determined whether the projected image is occluded by the target object. This allows for the determination of whether the target object detected in the first image captured in this current capture is the same as the target object detected in the first image captured in a previous capture, avoiding the misdetection of objects in the projected image as the target object and reducing the probability of false positives.

[0042] According to the first aspect, or any implementation of the first aspect above, the processing module is also used for:

[0043] When the difference information does not meet the first preset condition, the motion speed information of the target object in the third image is obtained; based on the motion speed information of the target object in the third image and the region description information of the area where the target object is located in the third image, the predicted region description information of the area of ​​the target object in the second image is determined; based on the predicted region description information of the area of ​​the target object in the second image and the region description information of the area where the projected image is located in the second image, it is determined whether the projected image is occluded by the target object. Thus, when it is determined that the target object detected in the first image captured this time is not the same object as the target object detected in the first image captured previously (i.e., when the object in the projected image is detected as the target object), the region description information of the area of ​​the target object detected in the first image captured previously can be predicted based on the region description information of the area where the target object was located in the first image captured previously.

[0044] According to the first aspect, or any implementation of the first aspect above, the processing module is used for:

[0045] The first image is processed using any one of a human detection algorithm, a human segmentation algorithm, or a depth estimation algorithm to obtain region description information of the area where the target object is located in the first image. Based on the region description information of the area where the target object is located in the first image, it is determined whether a second preset condition is met. When the second preset condition is met, based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, it is determined whether the projected image is occluded by the target object. In this way, it can be ensured that the target object detected from the first image is not an object in the projected image.

[0046] According to the first aspect, or any implementation of the first aspect above, the second preset condition includes that the area where the target object is located in the first image containing the target object does not overlap with the area where the projected image is located (or that the target object in the first image containing the target object is located outside the area where the projected image is located).

[0047] It should be understood that the second preset condition may also include other conditions, such as: the presence of a target object in two consecutive first images, the target object moving toward the projected image, or the target object's movement speed being within a preset range, at least one of these.

[0048] The first image containing the target object can be understood as the first image containing the target object among the first images acquired after the projection device is turned on and before the eye protection process is performed; or, it can be understood as the first image containing the target object among the first images acquired after the eye protection process is stopped and before the next eye protection process is performed.

[0049] According to the first aspect, or any implementation of the first aspect above, the processing module is used for:

[0050] The similarity between each image block in the preset region of the first image and the corresponding image block in the preset region of the reference image is determined; based on the region description information of the connected region composed of the target image blocks in the first image, the region description information of the region where the target object is located in the first image is determined, and the similarity between the target image block in the first image and the corresponding image block in the reference image is less than the similarity threshold.

[0051] Compared to human detection or human segmentation algorithms, determining the region description information of the target object in the first image based on similarity requires less computation.

[0052] Secondly, this application provides an image processing method, the method comprising: firstly, acquiring a first image, the first image being captured by an infrared fill light after being turned on and directed toward a projection medium, the projection medium displaying a projected image; subsequently, based on the first image, determining whether the projected image is occluded by a target object; then, when it is determined that the projected image is occluded by the target object, determining a first region of the projected image that is occluded and a second region in the source image that is occluded, the second region being the region in the source image corresponding to the first region in the projected image.

[0053] According to the second aspect, determining whether the projected image is occluded by a target object based on the first image includes: determining the region description information of the area where the target object is located in the first image based on the first image; obtaining the region description information of the area where the projected image is located in the first image; and determining whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image.

[0054] According to the second aspect, or any implementation of the second aspect above, the method further includes: determining the difference information between the region description information of the target object's location in the second image and the region description information of the target object's location in the third image, wherein the second image is the first image acquired this time, and the third image is the first image acquired previously; determining whether the projected image is occluded by the target object based on the region description information of the target object's location in the first image and the region description information of the projected image's location in the first image includes: when the difference information meets a first preset condition, determining whether the projected image is occluded by the target object based on the region description information of the target object's location in the second image and the region description information of the projected image's location in the second image.

[0055] According to the second aspect, or any implementation of the second aspect above, determining whether the projected image is occluded by the target object based on the region description information of the region where the target object is located in the first image and the region description information of the region where the projected image is located in the first image, further includes: when the difference information does not meet the first preset condition, obtaining the motion speed information of the target object in the third image; determining the predicted region description information of the region of the target object in the second image based on the motion speed information of the target object in the third image and the region description information of the region where the target object is located in the third image; and determining whether the projected image is occluded by the target object based on the predicted region description information of the region of the target object in the second image and the region description information of the region where the projected image is located in the second image.

[0056] According to the second aspect, or any implementation thereof, based on the first image, determining the region description information of the area where the target object is located in the first image includes: processing the first image using any one of a human detection algorithm, a human segmentation algorithm, or a depth estimation algorithm to obtain the region description information of the area where the target object is located in the first image; determining whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image includes: determining whether a second preset condition is met based on the region description information of the area where the target object is located in the first image; when the second preset condition is met, determining whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image.

[0057] According to the second aspect, or any implementation of the second aspect above, the second preset condition includes that the region where the target object is located in the first image containing the target object does not overlap with the region where the projected image is located.

[0058] According to the second aspect, or any implementation of the second aspect above, based on the first image, determining the region description information of the area where the target object is located in the first image includes: determining the similarity between each image block in a preset region in the first image and the corresponding image block in a preset region in a reference image; and determining the region description information of the area where the target object is located in the first image based on the region description information of the connected region formed by the target image blocks in the first image, wherein the similarity between the target image blocks in the first image and the corresponding image blocks in the reference image is less than a similarity threshold.

[0059] According to the second aspect, or any implementation of the second aspect above, a depth estimation algorithm is used to process the first image to obtain region description information of the region where the target object is located in the first image, including: using the depth estimation algorithm to determine the depth map of the first image; binarizing the depth map of the image based on a binarization threshold to obtain a binarized image; and determining the region description information of the region where the target object is located based on the region description information of the connected region composed of target pixels in the binarized image, wherein the pixel value of the target pixel in the binarized image is less than the binarization threshold.

[0060] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0061] Thirdly, this application provides a projection device, which includes a processing module, a complementary metal-oxide-semiconductor (CMOS) component, and a projection lens component. The CMOS component includes a CMOS module and an infrared fill lamp, and the filter component of the CMOS module allows infrared light to pass through.

[0062] The projection lens assembly is used to project a first source image onto a projection medium to display a first projected image on the projection medium;

[0063] The CMOS module is used to capture images, including a first image captured toward the projection medium after the infrared fill light is turned on.

[0064] The processing module is used to determine whether the first projected image is occluded by the target object based on the first image; when it is determined that the first projected image is occluded by the target object, it determines the first region of the first projected image that is occluded and the second region in the second source image that is occluded, wherein the second region is the region in the second source image corresponding to the first region in the first projected image.

[0065] The projection lens assembly is also used to project a second source image of a second region that is obscured onto a projection medium to display a second projected image on the projection medium, wherein a third region in the second projected image is obscured, and the third region is the same as the first region.

[0066] For example, the idea that the third region is the same as the first region can be understood as the third region having the same position and size as the first region.

[0067] Fourthly, this application provides a projection device, which includes: a memory, a processor, a complementary metal-oxide-semiconductor (CMOS) component, and a projection lens assembly. The CMOS component includes: a CMOS module and an infrared fill lamp. The filter component of the CMOS module allows infrared light to pass through. The memory is coupled to the processor. The memory stores program instructions, which, when executed by the processor, cause the projection device to perform the following steps:

[0068] Acquire a first image, which is a projected image captured by the CMOS component towards the projection medium after the infrared fill light is turned on, and the projection medium displays the projected image.

[0069] Based on the first image, determine whether the projected image is occluded by the target object;

[0070] When it is determined that the projected image is occluded by the target object, the first region of the projected image that is occluded and the second region in the source image that is occluded are determined. The second region is the region in the source image corresponding to the first region in the projected image.

[0071] Fifthly, this application provides a projection device, which includes:

[0072] The information acquisition module is used to acquire the first image, which is taken by shooting towards the projection medium after the infrared fill light is turned on, and the projection medium displays the projected image.

[0073] The occlusion detection module is used to determine whether the projected image is occluded by the target object based on the first image;

[0074] The eye protection processing module is used to determine the first region of the projected image that is occluded and the second region in the source image that is occluded when it is determined that the projected image is occluded by the target object. The second region is the region in the source image corresponding to the first region in the projected image.

[0075] For example, the projection device provided in this application may be part of a projection device or may be a projection device itself.

[0076] In a sixth aspect, this application provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, cause the processors to perform the methods in the second aspect or any possible implementation of the second aspect.

[0077] In a seventh aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the method of the second aspect or any possible implementation thereof.

[0078] Eighthly, this application provides a computer program product including computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the method in the second aspect or any possible implementation thereof.

[0079] In this embodiment, the projection device, computer-readable storage medium, computer program product or chip, etc., are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above. Attached Figure Description

[0080] Figure 1A is a schematic diagram of an application scenario 100 according to an embodiment of this application;

[0081] Figure 1B is a projection diagram of an embodiment of this application;

[0082] Figure 1C is a projection diagram of an embodiment of this application;

[0083] Figure 2A is a schematic diagram of the appearance of a projection device 110 according to an embodiment of this application;

[0084] Figure 2B is a schematic diagram of the appearance of a CMOS component 210 according to an embodiment of this application;

[0085] Figure 2C is a schematic diagram of a filter switching method according to an embodiment of this application;

[0086] Figure 2D is a system block diagram of a projection device 110 according to an embodiment of this application;

[0087] Figure 2E is a schematic diagram of the sensing functions that can be realized by a projection device 110 according to an embodiment of this application;

[0088] Figure 3A is a schematic diagram of an image processing procedure 300 according to an embodiment of this application;

[0089] Figure 3B is a schematic diagram of an eye protection process according to an embodiment of this application;

[0090] Figure 3C is a schematic diagram of another eye protection process according to an embodiment of this application;

[0091] Figure 4A is a schematic diagram of a portion of another image processing procedure 400 according to an embodiment of this application;

[0092] Figure 4B is a schematic diagram of another part of the image processing process 400 in another embodiment of this application;

[0093] Figure 4C is a schematic diagram of the region description information where the target object is located in a fourth image according to an embodiment of this application;

[0094] Figure 5A is a schematic diagram of a portion of another image processing procedure 500 according to an embodiment of this application;

[0095] Figure 5B is a schematic diagram of another part of another image processing procedure 500 in an embodiment of this application;

[0096] Figure 5C is a schematic diagram of a preset area according to an embodiment of this application;

[0097] Figure 5D is a schematic diagram of an embodiment of this application for dividing image blocks;

[0098] Figure 5E is a schematic diagram of a target connected region according to an embodiment of this application;

[0099] Figure 5F is a schematic diagram of another target connected region according to an embodiment of this application;

[0100] Figure 5G is a schematic diagram of another preset area according to an embodiment of this application;

[0101] Figure 6A is a schematic diagram of a portion of another image processing procedure 600 according to an embodiment of this application;

[0102] Figure 6B is a schematic diagram of another part of another image processing process 600 in an embodiment of this application;

[0103] Figure 7A is a schematic diagram of a portion of another image processing procedure 700 according to an embodiment of this application;

[0104] Figure 7B is a schematic diagram of another part of the image processing process 700 in another embodiment of this application;

[0105] Figure 8 is a schematic diagram of an image processing apparatus 800 according to an embodiment of this application;

[0106] Figure 9A is a schematic diagram of an autofocus process 900 according to an embodiment of this application;

[0107] Figure 9B is a schematic diagram of a preset focus image according to an embodiment of this application;

[0108] Figure 10A is a schematic diagram of an automatic trapezoidal correction process 1000 according to an embodiment of this application;

[0109] Figure 10B is a schematic diagram of a preset trapezoidal correction diagram according to an embodiment of this application;

[0110] Figure 10C is a schematic diagram of a trapezoidal correction process according to an embodiment of this application;

[0111] Figure 11A is a schematic diagram of an ambient light adaptive process 1100 according to an embodiment of this application;

[0112] Figure 11B is a schematic diagram of a preset ambient light adaptive map according to an embodiment of this application;

[0113] Figure 12 is a schematic diagram of an automatic obstacle avoidance process 1200 according to an embodiment of this application;

[0114] Figure 13A is a schematic diagram of an automatic screen entry process 1300 according to an embodiment of this application;

[0115] Figure 13B is a schematic diagram of a screen according to an embodiment of this application;

[0116] Figure 14 is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0117] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0118] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0119] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0120] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0121] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0122] In the embodiments of this application, the modules / components shown in the framework diagram (or structural diagram or system diagram) are merely examples of this application. The actual framework (or structure or system) may include more or fewer modules / components than those shown in the diagram, or may have different component configurations. Furthermore, the various components / modules shown in the diagrams may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0123] Figure 1A is a schematic diagram of an application scenario 100 according to an embodiment of this application.

[0124] In Figure 1A, user 130 can turn on the projection device 110 and select the video to be played; then, the projection device 110 can project the source image sequence (including multiple source images) corresponding to the video selected by user 130 onto the projection screen 120 to display the projected image sequence (including multiple projected images 140) on the projection screen 120 (this process is also the projection process); in this way, user 130 can watch the video.

[0125] For example, "source image" can be understood as the image to be projected by the projection device. "Projected image" can be understood as the image displayed on the projection medium after the source image is projected onto the projection medium by the projection device; the projected image can also be called the projected screen. Here, the projection medium can refer to the entity used to carry and display the projected image, and the projection medium includes, but is not limited to, screens, projection screens 120, special coating materials, water surfaces, and frosted surfaces.

[0126] During projection, user 130 may enter the projection area of ​​projection device 110 (the projection area can refer to the spatial range in which the projection device can project the source image, including the physical space between the projection device and the projection medium), for example, user 130 moves from position 1 to position 2; at this time, the light emitted by projection device 110 will be blocked by user 130, and correspondingly, the projected image 140 displayed on projection screen 120 will also be blocked by user 130. When the user is facing the projection device 110 directly or to the side, the light emitted by projection device 110 will enter the eyes of user 130, causing damage to the eyes of user 130.

[0127] Based on this, the projection device 110 of this application can perform eye protection processing when it detects that the user 130 has entered the projection area; in this way, when the user 130 enters the projection area of ​​the projection device 110, the damage of the light emitted by the projection device 110 to the eyes of the user 130 can be reduced.

[0128] Figure 1B is a projection diagram of an embodiment of this application.

[0129] Referring to Figure 1B, 160 is the source image and 140 is the projected image. The projection device 110 can determine, based on the image it captures, the region 150 of the projected image 140 that is occluded by the user 130 (region 150 is determined by the overlap between the region of the projected image 140 in the image captured by the projection device 110 and the actual region where the user 130 is located (i.e., the region covered by the pixels contained in the user 130), and a first mapping relationship; the first mapping relationship can be determined based on the distance between the projection device 110 and the projection medium (i.e., the shooting distance), shooting parameters, the pose of the projection device 110, etc.). Then, it occludes the region 170 in the source image 160 corresponding to region 150 (region 170 can be determined based on region 150 and a second mapping relationship, which can be determined based on the pose of the projection device 110, the distance between the projection device 110 and the projection medium (i.e., the projection distance), and the projection parameters of the projection device 110, etc.); for example, a black mask is added to region 170 in the source image 160. Among them, determining the region 150 of the projected image 140 that is occluded by the user 130, and the region 170 in the occluded source image 160 that corresponds to the region 150, is one way to implement eye protection processing.

[0130] Figure 1C is another projection diagram of an embodiment of this application.

[0131] Referring to Figure 1C, 160 is the source image, and 140 is the projected image. The projection device 110 can determine, based on the image it captures, the area 180 of the projected image 140 that is obscured by the user 130 (area 180 is determined based on the overlap between the bounding box of the area where the projected image 140 is located and the area where the user 130 is actually located in the image captured by the projection device 110, and a first mapping relationship). Then, it obscures the area 190 in the source image 160 corresponding to area 180 (wherein area 180 can be determined based on area 190 and a second mapping relationship); for example, it adds a black mask to area 190 in the source image 160. Determining the area 180 of the projected image 140 obscured by the user 130, and obscuring the area 190 in the source image 160 corresponding to area 180, is one implementation of eye protection processing.

[0132] Figure 2A is a schematic diagram of the appearance of a projection device 110 according to an embodiment of this application.

[0133] In Figure 2A, the projection device 110 may include a complementary metal-oxide-semiconductor (CMOS) component 210 and a projection lens component 220. It should be understood that the projection device 110 may also include a processing module and a storage module.

[0134] Figure 2B is a schematic diagram of the appearance of a CMOS component 210 according to an embodiment of this application.

[0135] For example, CMOS component 210 may include CMOS module 211 and infrared fill light 212. CMOS module 211 may include a lens, a CMOS sensor, and a filter structure.

[0136] In Figure 2B(1), the CMOS component 210 may include multiple infrared fill lights 212, which surround the lens of the CMOS module 211 and form a ring.

[0137] In Figure 2B(2), the CMOS component 210 may include multiple infrared fill lights 212, which surround the lens of the CMOS module 211 and form a rectangle.

[0138] In Figure 2B(3), the CMOS component 210 may include an infrared fill light 212, which is deployed side by side with the lens of the CMOS module 211.

[0139] It should be understood that this application does not limit the number and distribution of infrared fill lights 212 included in the CMOS component 210; wherein, the smaller the power of a single infrared fill light, the more infrared fill lights are included in the CMOS component 210.

[0140] In one possible embodiment, the CMOS module 211 may include a camera, which may include a lens, a CMOS sensor, and one or more filter structures. At least one of the one or more filter structures may allow infrared light to pass through. When the CMOS module 211 includes one filter structure, the filter structure may be a full-pass filter structure. When the CMOS module 211 includes two filter structures, the two filter structures may be a full-pass filter structure and an infrared cutoff filter structure, respectively; or, the two filter structures may be a narrowband filter structure and an infrared cutoff filter structure, respectively. When the CMOS module 211 includes three filter structures, the three filter structures may be a full-pass filter structure, a narrowband filter structure, and an infrared cutoff filter structure, respectively.

[0141] For example, the filter structure can be a filter coating or a filter. This application uses a filter structure as an example for illustration. When the CMOS module is a camera and the camera includes a filter, the CMOS module 211 can be regarded as a fixed filter camera, and the filter of the camera is an all-pass filter. When the CMOS module is a camera and the camera includes multiple filters, the camera can be called an adjustable (or switchable) filter camera.

[0142] Figure 2C is a schematic diagram of a filter switching method according to an embodiment of this application.

[0143] As shown in Figure 2C, this adjustable (or switchable) filter camera can switch the filter corresponding to the position of the lens in the CMOS module 211 by moving it left and right, moving it up and down, or rotating it. In this way, light can pass through the lens in the CMOS module 211 and the filter corresponding to that lens position and be incident on the CMOS sensor in the CMOS module 211. It should be understood that this application does not limit the method of adjusting (or switching) the filter.

[0144] In one possible approach, the CMOS module 211 may include multiple cameras, each of which includes a lens, a CMOS sensor, and a filter structure, with any two cameras having different filter structures. In this case, each of the multiple cameras is a fixed-filter camera. For example, the CMOS module 211 may include a camera that includes only a full-pass filter and a camera that includes only an infrared-cut filter (IR-CUT). As another example, the CMOS module 211 may include a camera that includes only a narrowband filter and a camera that includes only an infrared-cut filter (IR-CUT). Yet another example, the CMOS module 211 may include a camera that includes only a full-pass filter, a camera that includes only a narrowband filter, and a camera that includes only an infrared-cut filter.

[0145] It should be understood that the CMOS module 211 of this application may further include: a first camera and a second camera, wherein the first camera includes a lens, a CMOS sensor and multiple filter structures, and the second camera includes a lens, a CMOS sensor and a filter structure. That is, the CMOS module 211 may simultaneously include a camera with a fixed filter structure and a camera with an adjustable filter structure.

[0146] The following embodiments use CMOS module 211 as an example of an adjustable filter camera.

[0147] Figure 2D is a system block diagram of a projection device 110 according to an embodiment of this application.

[0148] In Figure 2D, the projection device 110 may include: an interaction module 21, a control module 22, a signal acquisition module 23, a storage module 24, a central processing module 25, and a projection module 26. The signal acquisition module 23 may include a CMOS component 210 and other sensing components; the projection module 26 may include a projection lens assembly 220. The aforementioned processing modules may include: the interaction module 21, the control module 22, the central processing module 25, and the projection module 26 (excluding the projection lens assembly 220).

[0149] The interaction module 21 can read and parse user-input interaction information, and then pass the parsed instructions to the control module. Users can input interaction information via remote control, gestures, voice, mobile applications, etc. The parsed instructions may include, but are not limited to: enabling ambient light adaptive function, enabling autofocus function, enabling automatic correction (or keystone correction) function, enabling automatic screen entry function, enabling automatic obstacle avoidance function, and enabling intelligent eye protection function.

[0150] The control module 22 can control the signal acquisition module 23 based on the instructions or system commands parsed by the interaction module 21. For example, when the instruction parsed by the interaction module 21 is to enable the autofocus function, the control module 22 is responsible for controlling the infrared fill light in the CMOS component 210 to turn off, switching the position of the IR-CUT filter and the lens of the CMOS module to correspond, and instructing the CMOS module to acquire visible light images. As another example, when the instruction parsed by the interaction module 21 is to enable the intelligent eye protection function, the control module 22 is responsible for controlling the infrared fill light in the CMOS component 210 to turn on, switching the position of the full-pass filter / narrow-band filter and the lens of the CMOS module to correspond, and instructing the CMOS module to acquire full-pass / narrow-band images.

[0151] The signal acquisition module 23 is used to acquire sensor data; for example, the CMOS module can be used to acquire visible light images, full-pass images or narrowband images.

[0152] The storage module 24 is used to store the sensing data obtained by the signal acquisition module 23 and output it to the central processing module 25; it is also used to store the cache information generated by the central processing module 25 (including cache data and processing results generated during algorithm operation).

[0153] The central processing module 25 is used to: 1) process the sensor data (or pre-processed sensor data) output by the storage module 24 using the algorithms corresponding to each sensing function to obtain the processing result; 2) transfer the data generated during the processing and required for subsequent processes to the storage module for storage; 3) process the source image based on the processing result and output the processed source image to the projection module 26.

[0154] The projection module 26 is used to call the optical engine to process the processed source image, and then project the processed source image onto the projection medium 120 through the projection lens assembly 220.

[0155] Figure 2E is a schematic diagram of the sensing functions that a projection device 110 according to an embodiment of this application can realize. In Figure 2E, the projection device 110 of this application equipped with a CMOS component 210 can realize sensing functions including but not limited to the following based on the CMOS component: 1. ambient light adaptive function, 2. autofocus function, 3. keystone correction function, 4. automatic obstacle avoidance function, 5. automatic screen entry function, 6. intelligent eye protection function.

[0156] For example, after one or more of the autofocus function, keystone correction function, automatic obstacle avoidance function, and automatic screen entry function are enabled, the projection device 110 may perform one or more of the following processes once during the power-on process: autofocus processing, keystone correction processing, automatic obstacle avoidance processing, and automatic screen entry processing. Subsequently, during projection, if the projection device 110 detects movement of the projection device 110 or receives interactive information input by the user, it may perform one or more of the following processes again: autofocus processing, keystone correction processing, automatic obstacle avoidance processing, and automatic screen entry processing; otherwise, none of the following processes are performed: autofocus processing, keystone correction processing, automatic obstacle avoidance processing, and automatic screen entry processing.

[0157] For example, when the ambient light adaptive function is enabled, the projection device 110 can perform ambient light adaptive processing in real time during projection. When the intelligent eye protection function is enabled, the projection device 110 can perform eye protection processing in real time during projection.

[0158] The following describes the process by which the projection device 110 implements various sensing functions.

[0159] Intelligent eye protection function

[0160] Figure 3A is a schematic diagram of an image processing procedure 300 according to an embodiment of this application. The image processing procedure 300 can be executed by the aforementioned central processing module 25.

[0161] S301, acquire the first image. The first image is captured by taking a picture of the projection medium after the infrared fill light is turned on, and the projection medium displays the projected image.

[0162] For example, when implementing any of the sensing functions such as autofocus, keystone correction, automatic obstacle avoidance, automatic screen entry, or environmental adaptation, the control module 22 can turn off the infrared fill light in the CMOS component and switch the position of the full-pass filter or IR-CUT filter to correspond with the lens position in the CMOS module; then, the control module 22 instructs the CMOS module to capture images in real time or periodically; subsequently, the CMOS module stores the captured images in the storage module 24. In implementing the intelligent eye protection function, the control module 22 can turn on the infrared fill light in the CMOS component and switch the position of the full-pass filter or narrowband filter to correspond with the lens position in the CMOS module; then, the control module 22 instructs the CMOS module to capture images in real time or periodically; subsequently, the CMOS module stores the captured images in the storage module 24. For ease of explanation, this application refers to the image captured by the CMOS module towards the projection medium after the infrared fill light is turned on (the projection device 110 typically projects the source image towards the projection medium; correspondingly, the CMOS module also captures images towards the projection medium) as the first image. It should be understood that when the infrared fill light is turned on, the CMOS module is directed toward the projection medium to take pictures. In addition to capturing the projection medium itself, it can also capture the projected image displayed on the projection medium.

[0163] For example, when a target object enters the projection area of ​​the projection device 110, the CMOS module takes a picture of the projection medium after the infrared fill light is turned on, and can also capture the target object. Then, the central processing module 25 can read the first image from the storage module 24; then, based on the first image, it determines whether eye protection processing is required, that is, it executes S302.

[0164] For example, the target object may include, but is not limited to, people, animals, etc.

[0165] S302, based on the first image, determine whether the projected image is occluded by the target object.

[0166] For example, the central processing module 25 can read one first image from the storage module 24 at a time, according to the order in which the first images were captured. After reading each first image from the storage module 24, the central processing module 25 can determine, based on that first image, whether the projected image is occluded by a target object. Since the projected image will be occluded by the target object when it enters the projection area, it is necessary to determine whether the projected image is occluded by the target object.

[0167] When it is determined that the projected image is obscured by the target object, it can be confirmed that the target object has entered the projection area. At this time, the central processing module 25 can execute S303, that is, perform eye protection processing. When it is determined that the projected image is not obscured by the target object, the central processing module 25 can return to execute S301, that is, read the next first image from the storage module 24.

[0168] For example, the central processing module 25 can determine the region description information of the area where the target object is located in the first image based on the first image; and obtain the region description information of the area where the projected image is located in the first image; and then, based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, determine whether the projected image is occluded by the target object.

[0169] For example, the central processing module 25 can first detect whether there is a target object in the first image; when it is determined that there is a target object in the first image, based on the detection results and / or other processing of the first image, it can determine the region description information of the area where the target object is located in the first image.

[0170] In one possible approach, the region where the target object is located can refer to the area covered by the pixels contained in the target object. In this case, the descriptive region information of the region where the target object is located can refer to the coordinates of the edge pixels of the area covered by the pixels contained in the target object.

[0171] In one possible approach, the region where the target object is located can refer to the region bounded by the outer bounding box of the area covered by the pixels contained in the target object. In this case, the descriptive region information of the region where the target object is located can refer to the descriptive information of the outer bounding box (such as the center coordinates of the outer bounding box (or the coordinates of the upper left corner of the outer bounding box), the height of the outer bounding box, and the width of the outer bounding box).

[0172] When it is determined that there is no target object in the first image, the central processing module 25 can return to execute S301, that is, read the next first image from the storage module 24.

[0173] In one possible approach, the central processing module 25 can identify the region description information of the area where the projected image is located in the first image based on the first image. The projected image is typically rectangular, and the region description information of the area where the projected image is located in the first image can include the center coordinates (or the coordinates of the top-left corner of the rectangle), the height of the rectangle, and the width of the rectangle.

[0174] In one possible approach, the central processing module 25 can read from the storage module 24 the area description information of the projected image region in the image captured by the CMOS, generated during the process of other previously implemented sensing functions (such as autofocus, keystone correction, automatic obstacle avoidance, and automatic screen entry).

[0175] In one possible approach, the central processing module 25 can read the region description information of the area where the projected image is located in the previously determined first image from the storage module 24.

[0176] For example, the central processing module 25 can determine whether the area where the target object is located in the first image overlaps with the area where the projected image is located in the first image based on the area description information of the area where the target object is located in the first image and the area description information of the area where the projected image is located in the first image.

[0177] When the central processing module 25 determines that the area where the target object is located in the first image overlaps with the area where the projected image is located in the first image, it determines that the projected image is occluded by the target object; at this time, S303 can be executed, that is, eye protection processing can be performed.

[0178] When the central processing module 25 determines that the area where the target object is located in the first image does not overlap with the area where the projected image is located in the first image, it determines that the projected image is not occluded by the target object. At this time, there is no need to perform eye protection processing, and it can return to execute S301, that is, read the next first image from the storage module 24.

[0179] S303, when it is determined that the projected image is occluded by the target object, the first region of the projected image that is occluded and the second region in the source image that is occluded are determined, and the second region is the region in the source image corresponding to the first region of the projected image.

[0180] For example, when it is determined that the projected image is occluded by a target object, the central processing module 25 can calculate the area of ​​the projected image occluded by the occluded target object (hereinafter referred to as the first area).

[0181] For example, the central processing module 25 can calculate the overlapping area between the region where the target object is located in the first image and the region where the projected image is located in the first image; then, the central processing module 25 can determine the first region based on the first mapping relationship and the overlapping area. Subsequently, the central processing module 25 can determine the region in the source image corresponding to the first region in the projected image (hereinafter referred to as the second region) based on the second mapping relationship; then, the central processing module 25 can occlude the second region in the source image.

[0182] For example, one way to occlude the second region in the source image is to add a black mask to the second region in the source image. It should be understood that other dark masks (dark colors are less harmful to the eyes than light colors) can also be added to the second region in the source image, and this application does not limit this.

[0183] In this way, after the projection lens assembly of the subsequent projection device 110 projects the source image that blocks the second area onto the projection medium, the second area in the source image will be projected onto the target object; since the second area in the source image is blocked, the harm of the light emitted by the projection device 110 to the eyes of the user 130 can be reduced.

[0184] Figure 3B is a schematic diagram of an eye protection process according to an embodiment of this application.

[0185] Figure 3B(1) shows the region where the projected image is located in the first image, and the bounding box of the region covered by the pixels of the target object in the first image. Based on the overlapping area of ​​the region where the projected image is located in the first image and the bounding box of the region covered by the pixels of the target object in the first image, and the first mapping relationship, the first region can be determined, as shown in Figure 3B(2). Then, based on the first region and the second mapping relationship, the second region in the source image can be determined; as shown in Figure 3B(3), a black mask is added to the second region of the source image. After the source image with the black mask added to the second region in Figure 3B(3) is projected onto the projection medium, the projected images displayed on the projection medium and the target object are shown in Figure 3B(4).

[0186] Figure 3C is a schematic diagram of another eye protection process according to an embodiment of this application.

[0187] Figure 3C(1) shows the region where the projected image is located in the first image, and the region covered by the pixels of the target object in the first image. Based on the overlapping area of ​​the region where the projected image is located in the first image and the region covered by the pixels of the target object in the first image, and the first mapping relationship, the first region can be determined, as shown in Figure 3C(2). Then, based on the first region and the second mapping relationship, the second region in the source image can be determined, as shown in Figure 3C(3) where a black mask is added to the second region of the source image. After the source image with the black mask added to the second region in Figure 3C(3) is projected onto the projection medium, the projected images displayed on the projection medium and the target object are shown in Figure 3C(4).

[0188] Since users typically watch movies using the projection device 110 in dimly lit environments, turning on the infrared fill light improves the imaging quality of the target object by the CMOS module. Furthermore, based on the image captured by the CMOS module (i.e., the first image), it can be accurately determined whether the target object has entered the projection area. After confirming that the target object has entered the projection area, eye protection processing is performed (i.e., identifying the first area of ​​the projected image that is obscured and the second area in the source image that is obscured). In other words, this application, based on a CMOS component equipped with an infrared fill light, can achieve intelligent eye protection functionality.

[0189] It should be noted that the source image corresponding to the projected image displayed on the projection medium in S301 and the source image in S303 can be the same image (corresponding to the scene of the projection device 110 projecting an image (such as the main interface)) or different images (corresponding to the scene of the projection device 110 projecting a video). This application does not impose any restrictions on this.

[0190] Figures 4A and 4B are schematic diagrams of another image processing procedure 400 according to an embodiment of this application. Image processing procedure 400 is based on image processing procedure 300 and describes an implementation method for determining whether a projected image is occluded by a target object based on a first image captured by a CMOS component.

[0191] S401, the control module switches the filter in the CMOS module corresponding to the lens position to a full-pass filter, turns on the infrared fill light, and instructs the CMOS module to take a picture.

[0192] For example, after the projection device 110 is powered on, the control module 22 can detect whether the intelligent eye protection function of the projection device 110 is enabled. When the intelligent eye protection function of the projection device 110 is detected to be enabled, the filter corresponding to the lens position in the CMOS module can be switched to an all-pass filter, and the infrared fill light can be turned on. Then, the control module 22 can send a control instruction to the CMOS module to instruct the CMOS module to take a picture.

[0193] In one possible scenario, the intelligent eye protection function of the projection device 110 is enabled by default when it leaves the factory.

[0194] In one possible scenario, the intelligent eye protection function of the projection device 110 is off by default when it leaves the factory. After the projection device 110 is turned on, the user can perform the operation to enable the intelligent eye protection function. The projection device 110 can respond to the user's operation and enable the intelligent eye protection function, that is, switch the state of the intelligent eye protection function to the on state.

[0195] For example, an all-pass filter can allow both visible and infrared light to pass through.

[0196] S402, the CMOS module takes a picture to obtain a first image and stores the first image in the storage module.

[0197] For example, the CMOS module can take a picture according to the control instructions sent by the control module 22 to obtain a first image.

[0198] In one possible approach, the control instructions sent by the control module 22 can be used to instruct the CMOS module to perform real-time shooting. In this way, the CMOS module can perform real-time shooting to obtain a first image.

[0199] In one possible approach, the control instruction sent by the control module 22 can be used to instruct the CMOS module to take pictures according to a preset period. In this way, the CMOS module can take pictures according to the preset period to obtain a first image. For example, the preset period can be set based on the duration during which the central processing module 25 determines whether the projected image is occluded by the target object based on the first image taken by the CMOS module; wherein the preset period can be less than or equal to this duration.

[0200] For example, the CMOS module can store the captured first image in the storage module 24.

[0201] S403, the central processing module reads a first image from the storage module to obtain the i-th fourth image, where the initial value of i is 1.

[0202] Where i is a positive integer.

[0203] For example, the central processing module 25 can read one first image from the storage module 24 each time according to the shooting time sequence of the first image, and then execute S404.

[0204] For example, the fourth image may refer to the first image read by the central processing module 25 from the storage module 24 after the projection device 110 (with its smart eye protection state enabled) is started and before the projection device 110 performs the first eye protection process, or after the projection device 110 stops the eye protection process and before the next eye protection process is performed.

[0205] S404, the central processing module determines whether the target object exists in the i-th fourth image.

[0206] For example, the central processing module 25 can use a human detection algorithm to perform human detection on the i-th fourth image to obtain a detection result; then, based on the detection result, it can determine whether there is a target object in the i-th fourth image. When the detection result is a detection box (that is, the bounding box of the area covered by the pixels contained in the target object), it can be determined that there is a target object in the i-th fourth image, and at this time, S406 can be executed. When the detection result is not a detection box, it can be determined that there is no target object in the i-th fourth image, and at this time, S405 can be executed.

[0207] S405, the central processing module increments i by 1.

[0208] For example, after S405 is executed, the process can return to execute S403, that is, the central processing module 25 can read the next first image from the storage module.

[0209] S406, the central processing module determines the motion state information of the target object in the i-th fourth image based on the detection results obtained by performing human detection on the i-th fourth image using a human detection algorithm.

[0210] For example, the motion state information of the target object in the i-th fourth image may include: region description information of the area where the target object is located in the i-th fourth image and motion velocity information of the target object in the i-th fourth image. In one possible embodiment, the motion velocity information of the target object in the i-th fourth image may include the motion velocity of the target object in the i-th fourth image. In another possible embodiment, the motion velocity information of the target object in the i-th fourth image may include: the initial velocity of the target object in the i-th fourth image and the acceleration of the target object in the i-th fourth image. This application uses the motion velocity of the target object in the i-th fourth image as an example for illustration.

[0211] In one possible scenario, the detection result obtained by human detection may include the coordinates of the top-left corner and the bottom-right corner of the detection box. The detection box obtained by human detection is shown in Figure 3B.

[0212] In one possible scenario, the detection results obtained from human detection may include the coordinates of the top-left corner of the detection box and the width and height of the detection box.

[0213] In one possible scenario, the detection results obtained from human detection may include the center coordinates of the detection box and the width and height of the detection box.

[0214] For example, the result of human detection performed on the i-th fourth image can be used as the region description information of the area where the target object is located in the i-th fourth image. This application uses the region description information of the area where the target object is located in the i-th fourth image, including the center coordinates of the detection box and the width and height of the detection box, as an example, as shown in Figure 4C.

[0215] Figure 4C is a schematic diagram of the region description information of the target object in a fourth image according to an embodiment of this application. In Figure 4C, the center coordinates of the detection box 0 of the (i-1)th fourth image are (x0, y0), the height of the detection box 0 is h0, and the width of the detection box 0 is w0. The center coordinates of the detection box 1 of the ith fourth image are (x1, y1), the height of the detection box 1 is h1, and the width of the detection box 1 is w1.

[0216] For example, when i = 1, that is, when the i-th fourth image is the first fourth image obtained by the central processing module 25 from the storage module 24, the motion speed of the target object in the i-th fourth image is 0. When i is greater than 1, that is, when the i-th fourth image is not the first fourth image obtained by the central processing module 25 from the storage module 24, the motion speed of the target object in the i-th fourth image can be determined based on the position of the target object in the i-th fourth image and the position of the target object in the (i-1)-th fourth image. The position of the target object in the fourth image can be represented by the center coordinates of the detection box. As shown in Figure 4B, the position of the target object in the (i-1)-th fourth image is (x0, y0), and the position of the target object in the i-th fourth image is (x1, y1); the motion speed of the target object in the i-th fourth image is v = x1 - x0.

[0217] For example, the central processing module 25 can also store the motion state information of the target object in the i-th fourth image into the storage module 24.

[0218] S407, the central processing module obtains the region description information of the area where the projected image is located in the i-th fourth image.

[0219] In one possible approach, the central processing module 25 can detect the region description information of the area where the projection screen or display screen is located in the i-th fourth image; and can determine the region description information of the area where the projection screen or display screen is located in the i-th fourth image as the region description information of the area where the projected image is located in the i-th fourth image.

[0220] In one possible approach, the central processing module 25 can use the region description information of the region where the projected image is located in the (i-1)th fourth image as the region description information of the region where the projected image is located in the i-th fourth image.

[0221] It should be understood that this application does not restrict the execution order of S406 and S407.

[0222] S408, the central processing module determines whether i is equal to 1.

[0223] If i equals 1, then proceed to step S405. If i does not equal 1, then proceed to step S409.

[0224] S409, the central processing module determines whether the second preset condition is met based on the region description information of the target object's location in one or more fourth images and the region description information of the projected image's location in the i-th fourth image.

[0225] For example, determining whether the second preset condition is met based on the region description information of the area where the target object is located in one or more fourth images can be understood as determining whether the second preset condition is met based on the region description information of the area where the target object is located in one or more fourth images from the first to the i-th images.

[0226] For example, to ensure that the target object detected in the i-th fourth image is not an object in the projected image, the second preset condition can be set as follows: the target object in the first fourth image containing the target object is located outside the area of ​​the projected image. For example, it can be determined whether the target object in the first fourth image containing the target object is located outside the area of ​​the projected image based on the area description information of the area where the target object is located in the first fourth image containing the target object and the area description information of the area where the projected image is located in the first fourth image containing the target object.

[0227] When the target object in the first fourth image containing the target object is located outside the area of ​​the projected image, it can be determined that the second preset condition is met, and S410 can be executed. When the target object in the first fourth image containing the target object is located within the projected image, it can be determined that the second preset condition is not met, and in this case, the execution can return to S405.

[0228] For example, the second preset condition may also include at least one of the following: the presence of a target object in two consecutive fourth images, the target object moving toward the projected image, or the movement speed of the target object being within a preset range.

[0229] For example, when the central processing module 25 determines that the storage module 24 stores the motion state information of the target object in the (i-1)th fourth image and the motion state information of the target object in the ith fourth image, it can determine that the target object exists in the two consecutive fourth images; otherwise, it determines that the target object does not exist in the two consecutive fourth images.

[0230] For example, the central processing module 25 can determine whether the target object is moving toward the projected image based on the motion state information of the target object in the (i-1)th fourth image and the motion state information of the target object in the i-th fourth image.

[0231] For example, the central processing module 25 can determine whether the motion speed of the target object is within a preset range based on the motion state information of the target object in the (i-1)th fourth image and the motion state information of the target object in the i-th fourth image.

[0232] If at least one of the following conditions is met: the target object exists in two consecutive fourth images, the target object moves toward the projected image, or the speed of the target object's movement is within a preset range, then the second preset condition is determined to be met, and S410 can be executed; otherwise, it can be determined that the second preset condition is not met, and S405 can be executed.

[0233] S410, the central processing module determines whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the i-th fourth image and the region description information of the area where the projected image is located in the i-th fourth image.

[0234] For example, the central processing module 25 can determine whether the region where the target object is located in the i-th fourth image overlaps with the region where the projected image is located, based on the region description information of the region where the target object is located in the i-th fourth image and the region description information of the region where the projected image is located in the i-th fourth image. When it is determined that the region where the target object is located in the i-th fourth image overlaps with the region where the projected image is located, it is determined that the projected image is occluded by the target object, and at this time, S411 can be executed. When it is determined that the region where the target object is located in the i-th fourth image does not overlap with the region where the projected image is located, it is determined that the projected image is not occluded by the target object, and S405 can be returned to be executed.

[0235] S411, the central processing module determines the first region of the projected image that is occluded and the second region in the occlusion source image based on the region description information of the target object region in the i-th fourth image and the region description information of the projected image region in the i-th fourth image.

[0236] For example, S411 is essentially an eye protection process, and S411 can be described with reference to the above description of S303, which will not be repeated here.

[0237] S412, the central processing module reads a first image from the storage module to obtain the kth fifth image, where the initial value of k is 1.

[0238] Where k is a positive integer.

[0239] For example, after S411 is executed, the central processing module 25 can execute S412, that is, starting from the i-th fourth image in the storage module 24, the central processing module 25 reads one first image at a time as the fifth image according to the shooting time sequence of the first images.

[0240] For example, the fifth image may refer to the first image obtained by the central processing module 25 from the storage module 24 after the projection device 110 has finished an eye protection process and before the eye protection process has stopped.

[0241] S413, the central processing module determines whether the target object exists in the k-th fifth image.

[0242] For example, S413 can be described with reference to the above description of S404, and will not be repeated here.

[0243] When it is determined that the target object exists in the k-th fifth image, the central processing module 25 may execute S414. When it is determined that the target object does not exist in the k-th fifth image, the central processing module 25 may execute S423.

[0244] S414, the central processing module obtains the region description information of the area where the projected image is located in the kth fifth image.

[0245] In one possible approach, the central processing module 25 can detect the region description information of the area where the projection screen or display screen is located in the kth fifth image; and can determine the region description information of the area where the projected image is located in the kth fifth image by using the region description information of the area where the projection screen or display screen is located in the kth fifth image.

[0246] In one possible approach, the central processing module 25 can use the region description information of the region where the projected image is located in the (k-1)th fifth image as the region description information of the region where the projected image is located in the kth fifth image.

[0247] S415, the central processing module determines the motion state information of the target object in the kth fifth image based on the detection results obtained by performing human detection on the kth fifth image using a human detection algorithm.

[0248] For example, the process of determining the motion state information of the target object in the kth fifth image in S414 can refer to the process of determining the motion state information of the target object in the ith fourth image in S406 above, and will not be repeated here.

[0249] It should be noted that when k equals 1, the velocity of the target object in the kth fifth image can be determined based on the position of the target object in the kth fifth image and the position of the target object in the ith fourth image. When k is greater than 1, the velocity of the target object in the kth fifth image can be determined based on the position of the target object in the (k-1)th fifth image and the position of the target object in the kth fifth image.

[0250] For example, this application does not limit the execution order of S414 and S415.

[0251] S416, the central processing module determines the difference information between the region description information of the target object's location in the k-th fifth image and the region description information of the target object's location in the (k-1)-th fifth image.

[0252] Among them, the k-th fifth image can also be called the second image, and the (k-1)-th fifth image can also be called the third image.

[0253] For example, when k is greater than 1, the central processing module 25 can determine the difference between the height of the region description information of the target object in the k-th fifth image and the height of the region description information of the target object in the (k-1)-th fifth image, thus obtaining a height difference. The central processing module 25 can also determine the difference between the width of the region description information of the target object in the k-th fifth image and the width of the region description information of the target object in the (k-1)-th fifth image, thus obtaining a width difference. Furthermore, the central processing module 25 can determine the distance between the center coordinates of the region description information of the target object in the k-th fifth image and the center coordinates of the region description information of the target object in the (k-1)-th fifth image, thus obtaining a positional difference.

[0254] For example, when k equals 1, the central processing module 25 can determine the difference between the height in the region description information of the target object's region in the k-th fifth image and the height in the region description information of the target object's region in the i-th fourth image, thus obtaining a height difference. The central processing module 25 can also determine the difference between the width in the region description information of the target object's region in the k-th fifth image and the width in the region description information of the target object's region in the i-th fourth image, thus obtaining a width difference. Furthermore, the central processing module 25 can determine the distance between the center coordinates in the region description information of the target object's region in the k-th fifth image and the center coordinates in the region description information of the target object's region in the i-th fourth image, thus obtaining a positional difference.

[0255] The difference information may include differences in height, width, and location.

[0256] S417, the central processing module determines whether the difference information meets the first preset condition.

[0257] For example, the first preset condition may include: the height difference is less than a height threshold, the width difference is less than a width threshold, and the position difference is less than a distance threshold. Therefore, when the height difference, width difference, and position difference in the difference information are all less than the height threshold, the width difference is less than the width threshold, and the position difference is less than the distance threshold, it can be determined that the difference information meets the first preset condition. This allows it to be determined that the target object detected in the first image captured this time is the same object as the target object detected in the first image captured previously, thus reducing the probability of false positives.

[0258] When the difference information meets the first preset condition, S418 can be executed; when the difference information does not meet the first preset condition, S419 can be executed.

[0259] S418, the central processing module determines whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the k-th fifth image and the region description information of the area where the projected image is located in the k-th fifth image.

[0260] For example, S418 can be described with reference to the above description of S410, and will not be repeated here.

[0261] When it is determined that the projected image is occluded by the target object, S421 can be executed; when it is determined that the projected image is not occluded by the target object, S424 can be executed.

[0262] S419, the central processing module determines the predicted region description information of the target object in the k-1 fifth image based on the motion state information of the target object in the k-1 fifth image.

[0263] For example, the central processing module 25 can determine the predicted motion speed of the target object in the k-th fifth image based on the motion speed of the target object in the (k-1)-th fifth image. For instance, the predicted motion speed of the target object in the k-th fifth image is equal to the motion speed of the target object in the (k-1)-th fifth image.

[0264] For example, the central processing module 25 can use the width and height of the detection box corresponding to the target object in the (k-1)th fifth image as the width and height of the predicted detection box corresponding to the target object in the (k-1)th fifth image; and use the ordinate of the center coordinate of the detection box corresponding to the target object in the (k-1)th fifth image as the ordinate of the center coordinate of the predicted detection box corresponding to the target object in the (k-1)th fifth image. The central processing module 25 can determine the abscissa of the center coordinate of the predicted detection box corresponding to the target object in the (k-1)th fifth image based on the motion speed and position of the target object in the (k-1)th fifth image. For example, the abscissa of the center coordinate of the predicted detection box corresponding to the target object in the (k-1)th fifth image = the abscissa of the center coordinate of the detection box corresponding to the target object in the (k-1)th fifth image + the motion speed of the target object in the (k-1)th fifth image.

[0265] The predicted region description information of the target object in the (k-1)th fifth image in the kth fifth image may include the height, width and center coordinates of the predicted detection box.

[0266] S420, the central processing module determines whether the projected image is occluded by the target object based on the predicted region description information of the region of the target object in the (k-1)th fifth image in the kth fifth image, and the region description information of the region where the projected image is located in the kth fifth image.

[0267] For example, S420 can be described with reference to the above description of S410, and will not be repeated here.

[0268] When it is determined that the projected image is occluded by the target object, S421 can be executed; when it is determined that the projected image is not occluded by the target object, S424 can be executed.

[0269] S421, the central processing module determines the first region of the projected image that is occluded and the second region in the occluded source image.

[0270] For example, S421 can be described with reference to S303 above, and will not be repeated here. After S421 is executed, S422 can be executed.

[0271] S422, the central processing module increments k by 1.

[0272] After S422 is executed, you can return to execute S412.

[0273] S423, the central processing module determines whether the target object in the (k-1)th fifth image is located at the edge of the area where the projected image is located in the (k-1)th fifth image.

[0274] For example, when k is not equal to 1, the central processing module 25 can determine whether the target object in the (k-1)th fifth image is located at the edge of the region where the projected image is located in the (k-1)th fifth image based on the region description information of the region where the target object is located. If it is determined that the target object in the (k-1)th fifth image is located at the edge of the region where the projected image is located in the (k-1)th fifth image, it means that it is reasonable that the target object was not detected in the kth fifth image, and S424 can be executed at this time. If it is determined that the target object in the (k-1)th fifth image is not located at the edge of the region where the projected image is located in the (k-1)th fifth image, it means that it is unreasonable that the target object was not detected in the kth fifth image, that is, the failure to detect the target object in the kth fifth image is a missed detection, and S419 can be executed at this time.

[0275] For example, when k equals 1, the (k-1)th fifth image refers to the ith fourth image. In this case, the central processing module 25 can determine whether the target object in the ith fourth image is located at the edge of the region where the projected image is located in the ith fourth image based on the region description information of the region where the target object is located in the ith fourth image. For details, please refer to the above description, which will not be repeated here.

[0276] S424, the central processing module sets i to 1 and k to 1.

[0277] It should be understood that when S424 is executed (when it is determined that there is no target object in the kth fifth image, or when the projected image is not occluded by the target object), the central processing module 25 stops the eye protection process, that is, the central processing module 25 does not occlude the second region in the source image.

[0278] After S424 is executed, you can return to execute S403 to monitor again whether eye protection treatment is needed.

[0279] It should be noted that when it is determined that there is no target object in the kth fifth image, or when the projected image is not occluded by the target object, the central processing module 25 can also send the source image of the unoccluded second area to the projection module 26; then, the projection module 26 can project the source image onto the projection medium 120.

[0280] Figure 5A is a schematic diagram of another image processing procedure 500 according to an embodiment of this application. Image processing procedure 500 is based on image processing procedure 300 and describes an implementation method for determining whether a projected image is occluded by a target object based on a first image captured by a CMOS component.

[0281] S501, the control module switches the filter in the CMOS module corresponding to the lens position to a narrowband filter, turns on the infrared fill light, and instructs the CMOS module to take a picture.

[0282] For example, S501 can be described with reference to S401 above, and will not be repeated here. It should be noted that the difference between S501 and S401 is that S501 switches the filter in the CMOS module corresponding to the lens position to a narrowband filter. Here, the narrowband filter in this application can refer to a filter that only allows infrared light to pass through, such as an 850nm narrowband filter (that is, it allows light with a wavelength of about 850 nanometers (nm) to pass through, while blocking or absorbing light of other wavelengths).

[0283] S502, the CMOS module takes a picture to obtain a first image and stores the first image in the storage module.

[0284] For example, S502 can be described with reference to the above description of S402, and will not be repeated here.

[0285] S503, the central processing module reads multiple consecutive first images from the storage module.

[0286] For example, the central processing module 25 can read multiple consecutive first images captured by the CMOS module within a preset time period from the storage module 24; wherein, the preset time period can be set as needed, such as 2 seconds, and this application does not limit it.

[0287] S504, the central processing module determines a reference image based on multiple consecutive first images.

[0288] For example, the central processing module 25 can calculate the similarity between any two first images in a series of consecutive first images; if the similarity between any two first images in a series of consecutive first images is greater than a preset value, it can be determined that the series of consecutive first images are all similar and that there is no target object in the series of consecutive first images; then, any one of the series of consecutive first images can be used as a reference image.

[0289] The algorithms for calculating the similarity between two first images may include, but are not limited to: histogram comparison method, histogram of oriented gradients (HOG) comparison method, mean squared error comparison method, and feature comparison method based on deep learning.

[0290] S505, the central processing module reads an image from the storage module to obtain the k-th fifth image, with the initial value of k being 1.

[0291] For example, the fifth image may refer to the first image captured by the CMOS module read from the storage module 24 after the reference image has been determined.

[0292] For example, the central processing module 25 can start from multiple first images in the storage module 24 and read one first image captured by the CMOS module each time, according to the shooting order of the first images, as the fifth image.

[0293] S506, the central processing module determines the similarity between each image block in the preset region of the kth fifth image and the corresponding image block in the preset region of the reference image.

[0294] For example, initially, when k=1, the preset region can refer to a pre-set initial region. Since the target object usually enters the projection region from both sides of the projected image, the initial region can be the region on both sides of the projected image in the fifth image, such as the region filled with dots in Figure 5C.

[0295] For example, the central processing module 25 can divide the k-th fifth image into multiple image blocks, and divide the reference image into multiple image blocks; wherein the k-th fifth image and the reference image are divided in the same way. As shown in FIG5D, the k-th fifth image (as shown in FIG5D(2)) and the reference image (as shown in FIG5D(1)) are both divided into multiple image blocks of the same size. Then, for each image block in the preset region of the k-th fifth image, the central processing module 25 can calculate the similarity between the image block and the corresponding image block in the preset region of the reference image.

[0296] S507, the central processing module determines whether a target object exists in the k-th fifth image based on the similarity between each image block in the preset region of the k-th fifth image and the corresponding image block in the preset region of the reference image.

[0297] For example, an image block in the k-th fifth image with a similarity less than a similarity threshold between it and the corresponding image block in the reference image can be referred to as a target image block (or a dissimilar image block). The similarity threshold can be set as needed.

[0298] For example, there may be multiple connected regions composed of target image patches (hereinafter referred to as the first connected region), and it can be determined whether the area of ​​the first connected region with the largest area is greater than an area threshold. The area threshold can be set as needed.

[0299] When the area of ​​the largest first connected region is less than or equal to the area threshold, it can be determined that there is no target object in the kth fifth image. At this time, S509 can be executed.

[0300] When the area of ​​the largest dissimilar connected region is greater than the area threshold, it can be determined that there is a target object in the k-th fifth image; at this time, S508 can be executed.

[0301] S508, the central processing module determines the region description information of the region where the target object is located in the k-th fifth image based on the region description information of the connected region composed of the target image block in the k-th fifth image; wherein, the similarity between the target image block in the k-th fifth image and the corresponding image block in the reference image is less than the similarity threshold.

[0302] For example, the central processing module 25 can obtain the target connected region by discarding one or more first connected regions and / or merging multiple first connected regions.

[0303] Specifically, the central processing module 25 can select the largest first connected region as the reference connected region; then, it calculates the distance between other first connected regions and the reference connected region. If the distance between another first connected region and the reference connected region is less than a preset value, the central processing module 25 can merge the other first connected region and the reference connected region; if the distance between another first connected region and the reference connected region is greater than or equal to the preset value, the central processing module 25 can discard the other first connected region.

[0304] Figure 5E is a schematic diagram of a target connected region according to an embodiment of this application.

[0305] In Figure 5E, there are 7 target image blocks. Among them, 6 target image blocks form connected region 2 (including the gray semi-transparent areas of the 6 target image blocks), and 1 target image block (noise image block) forms connected region 1 (including the gray semi-transparent area of ​​the 1 target image block). In this case, connected region 1 can be discarded; thus, the target connected region obtained is connected region 2.

[0306] Figure 5F is a schematic diagram of another target connected region according to an embodiment of this application.

[0307] In Figure 5F(1), there are 6 target image blocks. Among them, 2 target image blocks form connected region 1 (including the gray semi-transparent areas of the 2 target image blocks), and 4 target image blocks form connected region 2 (including the gray semi-transparent areas of the 4 target image blocks). Therefore, in this case, connected region 1 and connected region 2 can be merged to obtain connected region 3; thus, the obtained target connected region is connected region 3, as shown in Figure 5F(2).

[0308] Next, the region description information of the target connected region can be determined.

[0309] In one possible scenario, the region description information of the target connected region may include the coordinates of the top-left corner and the bottom-right corner of the target connected region.

[0310] In one possible scenario, the region description information of the target connected region may include the coordinates of the top-left corner of the target connected region and the width and height of the target connected region.

[0311] In one possible scenario, the region description information of the target connected region may include the center coordinates of the target connected region and the width and height of the target connected region.

[0312] Subsequently, based on the region description information of the target connected region, the region description information of the region where the target object is located in the k-th fifth image can be determined. For details, please refer to the description in S406 above regarding determining the region description information of the region where the target object is located in the i-th first image based on the detection results; it will not be repeated here.

[0313] S509, determine if k equals 1.

[0314] For example, if k equals 1, then S519 is executed. If k does not equal 1, then S520 is executed.

[0315] S510, the central processing module updates the preset region based on the region description information of the area where the target object is located in the kth fifth image.

[0316] For example, after determining the target connected region in S508, the central processing module 25 can update the preset region based on the region description information of the region where the target object is located in the k-th fifth image. The preset region includes the target connected region, and the preset region is larger than the target connected region.

[0317] Figure 5G is a schematic diagram of another preset area in an embodiment of this application.

[0318] Assuming that the target connected region is connected region 3 in Figure 5F(2), the preset region can be updated from the region filled with small black dots in Figure 5C to the region filled with small black dots in Figure 5G.

[0319] S511, the central processing module obtains the region description information of the area where the projected image is located in the kth fifth image.

[0320] For example, after S510 is executed, S511 can be executed.

[0321] S512, the central processing module determines whether k is equal to 1.

[0322] When k equals 1, S515 can be executed; when k does not equal 1, S513 can be executed.

[0323] S513, the central processing module determines the difference information between the region description information of the target object's location in the k-th fifth image and the region description information of the target object's location in the (k-1)-th fifth image.

[0324] S514, the central processing module determines whether the difference information meets the first preset condition.

[0325] When the difference information meets the first preset condition, S515 can be executed; when the difference information does not meet the first preset condition, S516 can be executed.

[0326] S515, the central processing module determines whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the kth fifth image and the region description information of the area where the projected image is located in the kth fifth image.

[0327] When it is determined that the projected image is occluded by the target object, S518 can be executed; when it is determined that the projected image is not occluded by the target object, S521 can be executed.

[0328] S516, the central processing module determines the predicted region description information of the target object in the k-1 fifth image based on the motion state information of the target object in the k-1 fifth image.

[0329] S517, the central processing module determines whether the projected image is occluded by the target object based on the predicted region description information of the region of the target object in the (k-1)th fifth image in the kth fifth image, and the region description information of the region where the projected image is located in the kth fifth image.

[0330] When it is determined that the projected image is occluded by the target object, S518 can be executed; when it is determined that the projected image is not occluded by the target object, S521 can be executed.

[0331] S518, the central processing module determines the first region of the projected image that is occluded and the second region in the occluded source image.

[0332] After S518 is executed, S519 will be executed.

[0333] S519, the central processing module increments k by 1.

[0334] After S519 is executed, you can return to execute S505.

[0335] S520, the central processing module determines whether the target object in the (k-1)th fifth image is located at the edge of the region where the projected image is located in the (k-1)th fifth image.

[0336] If it is determined that the target object in the (k-1)th fifth image is not located at the edge of the region where the projected image is located in the (k-1)th fifth image, it means that it is unreasonable that the target object was not detected in the kth fifth image. In other words, the failure to detect the target object in the kth fifth image is a missed detection. At this time, S516 can be executed.

[0337] If it is determined that the target object in the (k-1)th fifth image is located at the edge of the region where the projected image of the (k-1)th fifth image is located, then it is reasonable that the target object was not detected in the kth fifth image, and S521 can be executed at this time.

[0338] S521, the central processing module sets k to 1 and initializes the preset area.

[0339] For example, S513 to S521 can be described with reference to S416 to S424 above, and will not be repeated here. After S521 is executed, S505 can be executed.

[0340] Compared to human detection or human segmentation algorithms, determining the region description information of the target object in the first image based on similarity requires less computation.

[0341] Figures 6A and 6B are schematic diagrams of another image processing procedure 600 according to an embodiment of this application. Image processing procedure 600 is based on image processing procedure 300 and describes an implementation method for determining whether a projected image is occluded by a target object based on a first image captured by a CMOS component.

[0342] S601, the control module switches the filter in the CMOS module corresponding to the lens position to a narrowband filter, turns on the infrared fill light, and instructs the CMOS module to take a picture.

[0343] For example, S601 can be described with reference to the above description of S501, and will not be repeated here.

[0344] S602, the CMOS module takes a picture to obtain a first image and stores the first image in the storage module.

[0345] S603, the central processing module reads a first image from the storage module to obtain the i-th fourth image, where the initial value of i is 1.

[0346] For example, S602 to S603 can be referred to the description of S402 to S403 above, and will not be repeated here.

[0347] S604, the central processing module uses a depth estimation algorithm to determine the depth map of the i-th fourth image.

[0348] For example, the central processing module 25 can use a monocular depth estimation algorithm to process the i-th fourth image to determine the depth map of the i-th fourth image. The depth map of the i-th fourth image includes the distance information from each pixel in the i-th fourth image to the CMOS module.

[0349] S605, the central processing module binarizes the depth map of the i-th fourth image based on the binarization threshold to obtain the i-th binarized image.

[0350] It should be understood that the distances from the pixels in the projected image of the i-th fourth image to the CMOS module are the same, and the distances from the pixels in the projected image of the i-th fourth image to the CMOS module are greater than the distances from the pixels in the target object of the i-th fourth image to the CMOS module. Therefore, the pixel values ​​of each pixel in the i-th fourth image can be compared with a binarization threshold to binarize the depth map of the i-th fourth image; thus, based on the obtained binarized image, the target object in the i-th fourth image can be distinguished from the object in the projected image.

[0351] In one possible approach, the pixel values ​​of pixels in the i-th fourth image that are greater than or equal to the binarization threshold can be set to 1, and the pixel values ​​of pixels in the i-th fourth image that are less than the binarization threshold can be set to 0. In this way, the pixels containing the target object in the resulting binarized image will be black, and the other pixels will be white.

[0352] In one possible approach, the pixel values ​​of pixels in the i-th fourth image that are greater than or equal to the binarization threshold can be set to 0, and the pixel values ​​of pixels in the i-th fourth image that are less than the binarization threshold can be set to 1. In this way, the pixels containing the target object in the resulting binarized image will be white, and the other pixels will be black.

[0353] S606, the central processing module determines whether a target object exists in the i-th fourth image based on the i-th binarized image.

[0354] For example, the central processing module can determine whether the area of ​​the connected region formed by the target pixels in the binarized image is greater than the area threshold.

[0355] In one possible approach, if the area of ​​the connected region formed by the target pixels in the binarized image is greater than an area threshold, it can be determined that a target object exists in the i-th fourth image, and step S608 can be executed. If the area of ​​the connected region formed by the target pixels in the binarized image is less than or equal to the area threshold, it can be determined that no target object exists in the i-th fourth image, and step S607 can be executed.

[0356] In one possible approach, when the area of ​​the connected region formed by the target pixels in the binarized image is greater than an area threshold, and the shape of the connected region formed by the target pixels in the binarized image is similar to the shape of a human body (for example, the aspect ratio of the connected region formed by the target pixels in the binarized image can be calculated; if the aspect ratio is within a preset range, then it is determined that the shape of the connected region formed by the target pixels in the binarized image is similar to the shape of a human body), it can be determined that a target object exists in the i-th fourth image, and S608 can be executed. When the area of ​​the connected region formed by the target pixels in the binarized image is less than or equal to the area threshold, or the shape of the connected region formed by the target pixels in the binarized image is not similar to the shape of a human body, it can be determined that no target object exists in the i-th fourth image, and S607 can be executed.

[0357] It should be noted that the area threshold involved in S606 may be the same as or different from the area threshold involved in S507 above, and this application does not impose any restrictions on this.

[0358] In this case, the pixel value of the target pixel in the i-th binarized image is less than the binarization threshold.

[0359] S607, the central processing module increments i by 1.

[0360] For example, after executing S607, you can return to execute S603.

[0361] S608, the central processing module determines the region description information of the region where the target object is located based on the region description information of the connected region composed of the target pixels in the i-th binarized image.

[0362] In one possible approach, the region description information of the connected region formed by the target pixels in the i-th binarized image can be used as the region description information of the region where the target object is located in the i-th fourth image. In this case, the region description information of the region where the target object is located in the i-th fourth image is the coordinates of the contour pixels of the region covered by the pixels contained in the target object in the i-th fourth image.

[0363] In one possible approach, the region description information of the bounding box of the connected region formed by the target pixels in the i-th binarized image can be used as the region description information of the region where the target object is located in the i-th fourth image. In another possible scenario, the region description information of the connected region formed by the target pixels in the i-th binarized image may include the coordinates of the upper-left corner and the lower-right corner of the bounding box (the bounding box of the connected region formed by the target pixels in the i-th binarized image). In yet another possible scenario, the region description information of the connected region formed by the target pixels in the i-th binarized image may include the coordinates of the upper-left corner of the bounding box and the width and height of the bounding box. Finally, the region description information of the connected region formed by the target pixels in the i-th binarized image may include the center coordinates of the bounding box and the width and height of the bounding box.

[0364] When k is 1, the velocity of the target object in the i-th fourth image is 0. When k is not equal to 1, the velocity of the target object in the i-th fourth image can be determined based on the position of the target object in the i-th fourth image and the position of the target object in the (i-1)-th fourth image; for details, please refer to the description in S406 above, which will not be repeated here.

[0365] S609, the central processing module obtains the region description information of the area where the projected image is located in the i-th fourth image.

[0366] It should be understood that this application does not restrict the execution order of S608 and S609.

[0367] S610, determine if i is equal to 1.

[0368] If i equals 1, then proceed to S607. If i does not equal 1, then proceed to S611.

[0369] S611, the central processing module determines whether the second preset condition is met based on the region description information of the area where the target object is located in one or more fourth images and the region description information of the area where the projected image is located in the i-th fourth image.

[0370] If the second preset condition is met, S612 can be executed. If the second preset condition is not met, the process can return to executing S607.

[0371] S612, the central processing module determines whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the i-th fourth image and the region description information of the area where the projected image is located in the i-th fourth image.

[0372] When it is determined that the region where the target object is located in the i-th fourth image overlaps with the region where the projected image is located, it is determined that the projected image is occluded by the target object. At this time, S613 can be executed. When it is determined that the region where the target object is located in the i-th fourth image does not overlap with the region where the projected image is located, it is determined that the projected image is not occluded by the target object. The process can then return to execute S607.

[0373] S613, the central processing module determines the first region of the projected image that is occluded and the second region in the occlusion source image based on the region description information of the target object region in the i-th fourth image and the region description information of the projected image region in the i-th fourth image.

[0374] S614, the central processing module reads a first image from the storage module to obtain the kth fifth image, with the initial value of k being 1.

[0375] S615, the central processing module uses a depth estimation algorithm to determine the depth map of the k-th fifth image.

[0376] S616, the central processing module binarizes the depth map of the kth fifth image based on the binarization threshold to obtain the kth binarized image.

[0377] S617, the central processing module determines whether the target object exists in the kth binarized image based on the kth image.

[0378] For example, S615 to S617 can be described with reference to the above description of S604 to S606, and will not be repeated here.

[0379] When it is determined that the target object exists in the k-th fifth image, the central processing module 25 may execute S618. When it is determined that the target object does not exist in the k-th fifth image, the central processing module 25 may execute S627.

[0380] S618, the central processing module obtains the region description information of the area where the projected image is located in the kth fifth image.

[0381] S619, the central processing module determines the region description information of the region where the target object is located based on the region description information of the connected region composed of the target pixels in the k-th binarized image.

[0382] It should be understood that this application does not restrict the execution order of S618 and S619.

[0383] S620, the central processing module determines the difference information between the region description information of the target object's location in the k-th fifth image and the region description information of the target object's location in the (k-1)-th fifth image.

[0384] S621, the central processing module determines whether the difference information meets the first preset condition.

[0385] When the difference information meets the first preset condition, S622 can be executed; when the difference information does not meet the first preset condition, S623 can be executed.

[0386] S622, the central processing module determines whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the kth fifth image and the region description information of the area where the projected image is located in the kth fifth image.

[0387] When it is determined that the projected image is occluded by the target object, S625 can be executed; when it is determined that the projected image is not occluded by the target object, S628 can be executed.

[0388] S623, the central processing module determines the predicted region description information of the target object in the k-1 fifth image based on the motion state information of the target object in the k-1 fifth image.

[0389] S624, the central processing module determines whether the projected image is occluded by the target object based on the predicted region description information of the region of the target object in the (k-1)th fifth image in the kth fifth image, and the region description information of the region where the projected image is located in the kth fifth image.

[0390] When it is determined that the projected image is occluded by the target object, S625 can be executed; when it is determined that the projected image is not occluded by the target object, S628 can be executed.

[0391] S625, the central processing module determines the first region of the projected image that is occluded and the second region in the occluded source image.

[0392] After S625 is executed, S626 can be executed.

[0393] S626, the central processing module increments k by 1.

[0394] After S626 is executed, you can return to execute S614.

[0395] S627, the central processing module determines whether the target object in the (k-1)th fifth image is located at the edge of the area where the projected image is located in the (k-1)th fifth image.

[0396] If it is determined that the target object in the (k-1)th fifth image is not located at the edge of the region where the projected image is located in the (k-1)th fifth image, it means that it is unreasonable that the target object was not detected in the kth fifth image. In other words, the failure to detect the target object in the kth fifth image is a missed detection. At this time, S623 can be executed.

[0397] If it is determined that the target object in the (k-1)th fifth image is located at the edge of the region where the projected image of the (k-1)th fifth image is located, then it is reasonable that the target object was not detected in the kth fifth image, and S628 can be executed at this time.

[0398] In S628, the central processing module sets i to 1 and k to 1.

[0399] For example, S618 to S628 can be described with reference to the above description of S414 to S422, and will not be repeated here.

[0400] After S628 is executed, you can return to execute S603 to monitor again whether eye protection processing is required.

[0401] Figures 7A and 7B are schematic diagrams of another image processing procedure 700 according to an embodiment of this application. Image processing procedure 700 is based on image processing procedure 300 and describes an implementation method for determining whether a projected image is occluded by a target object based on a first image captured by a CMOS component.

[0402] S701, the control module switches the filter in the CMOS module corresponding to the lens position to a narrowband filter, turns on the infrared fill light, and instructs the CMOS module to take a picture.

[0403] For example, S701 can be described with reference to the above description of S501, and will not be repeated here.

[0404] S702, the CMOS module takes a picture to obtain a first image and stores the first image in the storage module.

[0405] S703, the central processing module reads a first image from the storage module to obtain the i-th fourth image, where the initial value of i is 1.

[0406] S704, the central processing module determines whether the target object exists in the i-th fourth image.

[0407] For example, when it is determined that the target object exists in the i-th fourth image, S706 can be executed; when it is determined that the target object does not exist in the i-th fourth image, S705 can be executed.

[0408] For example, S702 to S704 can be referred to the description of S402 to S404 above, and will not be repeated here.

[0409] S705, the central processing module increments i by 1.

[0410] For example, after S705 is executed, the process can return to execute S703, that is, the central processing module 25 can read the next first image from the storage module.

[0411] S706, the central processing module determines the motion state information of the target object in the i-th fourth image based on the detection results obtained by performing human figure segmentation on the i-th fourth image using a human figure segmentation algorithm.

[0412] For example, the detection results obtained from human figure segmentation include the coordinates of the contour pixels of the area covered by the pixels contained in the target object.

[0413] For example, the result of human figure segmentation of the i-th first image can be used as the region description information of the area where the target object is located in the i-th first image.

[0414] For example, when i = 1, that is, when the i-th fourth image is the first fourth image obtained by the central processing module 25 from the storage module 24, the motion speed of the target object in the i-th fourth image is 0. When i is greater than 1, that is, when the i-th fourth image is not the first fourth image obtained by the central processing module 25 from the storage module 24, the motion speed of the target object in the i-th fourth image can be determined based on the position of the target object in the i-th fourth image and the position of the target object in the (i-1)-th fourth image. The position of the target object in the fourth image can be represented by the center coordinates determined by the coordinates of the contour pixels of the area covered by the pixels contained in the target object. The motion speed v of the target object in the i-th fourth image is equal to the abscissa of the center coordinates of the target object in the i-th fourth image minus the abscissa of the center coordinates of the target object in the (i-1)-th fourth image.

[0415] S707, the central processing module obtains the region description information of the area where the projected image is located in the i-th fourth image.

[0416] For example, this application does not limit the execution order of S706 and S707.

[0417] S708, the central processing module determines whether i is equal to 1.

[0418] If i equals 1, then proceed to step S705. If i does not equal 1, then proceed to step S709.

[0419] S709, the central processing module determines whether the second preset condition is met based on the region description information of the area where the target object is located in one or more fourth images and the region description information of the area where the projected image is located in the i-th fourth image.

[0420] If the second preset condition is met, S710 can be executed. If the second preset condition is not met, the process can return to executing S705.

[0421] S710, the central processing module determines whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the i-th fourth image and the region description information of the area where the projected image is located in the i-th fourth image.

[0422] If it is determined that the projected image is occluded by the target object, then S711 can be executed. If it is determined that the projected image is not occluded by the target object, then S705 can be executed.

[0423] S711, the central processing module determines the first region of the projected image that is occluded and the second region in the occlusion source image based on the region description information of the target object region in the i-th fourth image and the region description information of the projected image region in the i-th fourth image.

[0424] S712, the central processing module reads a first image from the storage module to obtain the kth fifth image, with the initial value of k being 1.

[0425] S713, the central processing module determines whether the target object exists in the k-th fifth image.

[0426] When it is determined that the target object exists in the k-th fifth image, the central processing module 25 may execute S714. When it is determined that the target object does not exist in the k-th fifth image, the central processing module 25 may execute S723.

[0427] S714, the central processing module obtains the region description information of the area where the projected image is located in the kth fifth image.

[0428] S715, the central processing module determines the motion state information of the target object in the kth fifth image based on the detection results obtained by performing human figure segmentation on the kth fifth image using a human figure segmentation algorithm.

[0429] It should be understood that this application does not restrict the execution order of S714 and S715.

[0430] S716, the central processing module determines the difference between the region description information of the target object's location in the k-th fifth image and the region description information of the target object's location in the (k-1)-th fifth image.

[0431] S717, the central processing module determines whether the difference information meets the first preset condition.

[0432] When the difference information meets the first preset condition, S718 can be executed; when the difference information does not meet the first preset condition, S719 can be executed.

[0433] S718, the central processing module determines whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the kth fifth image and the region description information of the area where the projected image is located in the kth fifth image.

[0434] When it is determined that the projected image is occluded by the target object, S721 can be executed; when it is determined that the projected image is not occluded by the target object, S724 can be executed.

[0435] S719, the central processing module determines the predicted region description information of the target object in the k-1 fifth image based on the motion state information of the target object in the k-1 fifth image.

[0436] S720, the central processing module determines whether the projected image is occluded by the target object based on the predicted region description information of the region of the target object in the (k-1)th fifth image in the kth fifth image, and the region description information of the region where the projected image is located in the kth fifth image.

[0437] When it is determined that the projected image is occluded by the target object, S721 can be executed; when it is determined that the projected image is not occluded by the target object, S724 can be executed.

[0438] S721, the central processing module determines the first region of the projected image that is occluded and the second region in the occluded source image.

[0439] After S721 is executed, S722 can be executed.

[0440] In S722, the central processing module increments k by 1.

[0441] After S722 is executed, you can return to execute S712.

[0442] S723, the central processing module determines whether the target object in the (k-1)th fifth image is located at the edge of the region where the projected image is located in the (k-1)th fifth image.

[0443] If it is determined that the target object in the (k-1)th fifth image is not located at the edge of the region where the projected image is located in the (k-1)th fifth image, it means that it is unreasonable that the target object was not detected in the kth fifth image. In other words, the failure to detect the target object in the kth fifth image is a missed detection. At this time, S719 can be executed.

[0444] If it is determined that the target object in the (k-1)th fifth image is located at the edge of the region where the projected image of the (k-1)th fifth image is located, then it is reasonable that the target object was not detected in the kth fifth image, and S724 can be executed at this time.

[0445] In S724, the central processing module sets i to 1 and k to 1.

[0446] After S724 is executed, you can return to execute S703 to monitor again whether eye protection processing is needed.

[0447] For example, S707 to S724 can be referred to the description of S407 to S424 above, and will not be repeated here.

[0448] It should be noted that the human segmentation algorithm in the graphics processing procedure 700 can also be replaced by the human detection algorithm described above.

[0449] This application embodiment also provides an image processing apparatus, which includes a processing module and a complementary metal-oxide-semiconductor (CMOS) component. The CMOS component includes a CMOS module and an infrared fill light, and the filter structure in the CMOS module allows infrared light to pass through.

[0450] The CMOS module is used to capture images, including a first image captured towards the projection medium after the infrared fill light is turned on, and the projection medium displays the projected image.

[0451] The processing module is used to determine whether the projected image is occluded by the target object based on the first image; when it is determined that the projected image is occluded by the target object, it determines the first region of the projected image that is occluded and the second region in the source image that is occluded, wherein the second region is the region in the source image corresponding to the first region in the projected image.

[0452] The processing module in the image processing device may be the central processing module 25 mentioned above.

[0453] Figure 8 is a schematic diagram of an image processing apparatus 800 according to an embodiment of this application. The image processing apparatus 800 can be used to perform the methods of the foregoing embodiments. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0454] For example, the image processing apparatus 800 may include:

[0455] The information acquisition module 801 is used to acquire a first image, which is captured by the infrared fill light after it is turned on and the projection medium displays the projected image.

[0456] The occlusion determination module 802 is used to determine whether the projected image is occluded by the target object based on the first image;

[0457] The eye protection processing module 803 is used to determine the first region of the projected image that is occluded and the second region in the source image that is occluded when it is determined that the projected image is occluded by the target object. The second region is the region in the source image corresponding to the first region in the projected image.

[0458] It should be noted that the information acquisition module 801, the occlusion judgment module 802, and the eye protection processing module 803 belong to the aforementioned central processing module 25.

[0459] This application also provides a projection device, which includes: a memory, a processor, a complementary metal-oxide-semiconductor (CMOS) component, and a projection lens assembly. The CMOS component includes: a CMOS module and an infrared fill lamp. The filter component of the CMOS module allows infrared light to pass through. The memory is coupled to the processor. The memory stores program instructions, which, when executed by the processor, cause the projection device to perform the following steps:

[0460] The first image is acquired by the CMOS component after the infrared fill light is turned on and it is directed toward the projection medium. The projection medium displays the projected image.

[0461] Based on the first image, determine whether the projected image is occluded by the target object;

[0462] When it is determined that the projected image is occluded by the target object, the first region of the projected image that is occluded and the second region in the source image that is occluded are determined. The second region is the region in the source image corresponding to the first region in the projected image.

[0463] The memory is an example of the aforementioned storage module 24, and the processor is an example of the aforementioned central processing module 25.

[0464] Autofocus function

[0465] Figure 9A is a schematic diagram of an autofocus process 900 according to an embodiment of this application.

[0466] S901, the control module switches the filter in the CMOS module corresponding to the lens position to an IR-CUT / all-pass filter and turns on the infrared fill light.

[0467] For example, the reason for using an IR-CUT or all-pass filter in implementing autofocus is that the projected image content needs to be clearly captured during the autofocus process.

[0468] Among them, IR-CUT can block near-infrared light, prevent the CMOS sensor from being overly sensitive to red light, and avoid color shift.

[0469] S902, the projection module uses different projection focal lengths to project a preset focus image onto the projection medium.

[0470] To achieve focusing, the projection device 110 needs to accurately analyze the sharpness of the projected image content. To achieve this, the projection module 26 typically projects a preset focus image, which has high contrast and sharpness, and clear texture and detail information; an example of a preset focus image can be shown in Figure 9B.

[0471] For example, the control module 22 can drive a stepper motor to adjust the focal length (i.e., projection focal length) of the lens in the projection lens assembly to project projection images corresponding to different focal lengths onto the projection medium.

[0472] The S903 uses a CMOS module to capture multiple sixth images and stores them in the storage module.

[0473] For example, the control module 22 drives the stepper motor to adjust the focal length of the lens in the projection lens assembly, while instructing the CMOS module to capture projection images corresponding to different projection focal lengths, thereby obtaining multiple sixth images and storing the sixth images and their corresponding projection focal lengths in the storage module 24.

[0474] S904, the central processing module reads multiple sixth images from the storage module.

[0475] S905, the central processing module determines the sharpness of each of the multiple sixth images.

[0476] For example, the sharpness of each sixth image can be represented by at least one of the following: contrast, image entropy, histogram, gradient strength, etc. Then, the sharpest sixth image and its corresponding projection focal length are stored in storage module 24.

[0477] S906, the control module sets the focal length of the lens in the projection lens assembly to the projection focal length corresponding to the clearest sixth image.

[0478] For example, the control module 22 adjusts the focal length of the lens in the projection lens assembly to the projection focal length corresponding to the clearest sixth image; subsequently, the projection module maps the image to be projected onto the projection medium by adjusting the projection focal length of the projection lens assembly, thus completing automatic focusing.

[0479] Automatic keystone correction function

[0480] Figure 10A is a schematic diagram of an automatic trapezoidal correction process 1000 according to an embodiment of this application.

[0481] S1001, the control module switches the filter in the CMOS module corresponding to the lens position to an IR-CUT / all-pass filter and turns on the infrared fill light.

[0482] S1002, the projection module projects a preset trapezoidal correction diagram onto the projection medium.

[0483] To achieve keystone correction, the projection device 110 needs to detect the coordinates of the four vertices of the projected image. To achieve this, a preset keystone correction image with clear edge information is typically projected using the projection module. An example of a preset keystone correction image is shown in Figure 10B. The preset keystone correction image in Figure 10B has clear edge information, making it easy to extract the coordinates of the four vertices of the projected image.

[0484] S1003, the CMOS module takes a picture, obtains the seventh image, and stores it in the storage module.

[0485] S1004, the central processing module reads the seventh image from the storage module.

[0486] S1005, the central processing module detects the vertex coordinates of the projected image in the seventh image.

[0487] For example, the central processing module can use corner detection algorithms (such as Harris corner detection) and edge detection algorithms (such as Canny and Sobel operators) to find clear edges in the projected image; then, it can filter quadrilateral regions by rules such as area and angle to obtain the coordinates of the four vertices of the projected image in the seventh image.

[0488] S1006, the central processing module performs perspective transformation on the vertex coordinates of the projected image in the seventh image to correct the positions of the four vertices of the projected image.

[0489] For example, after obtaining the coordinates of the four vertices of the projection region in the seventh image, the central processing module 25 can calculate the homography matrix; then, based on the perspective transformation and the homography matrix, it transforms the trapezoidal region into a rectangular region, thereby achieving automatic trapezoidal correction.

[0490] Figure 10C is a schematic diagram of a trapezoidal correction process according to an embodiment of this application. In Figure 10B, the dashed area is the projected image before trapezoidal correction, and the rectangular area is the projected image after trapezoidal correction. For example, the coordinates of the corrected projected image can be saved in a storage module for use in implementing other sensing functions.

[0491] Ambient light adaptive function

[0492] Figure 11A is a schematic diagram of an ambient light adaptive process 1100 according to an embodiment of this application.

[0493] S1101, the control module switches the filter in the CMOS module corresponding to the lens position to an IR-CUT / all-pass filter and turns on the infrared fill light.

[0494] S1102, the projection module projects a preset ambient light adaptive map onto the projection medium.

[0495] To achieve ambient light adaptation, the projection device 110 needs to accurately estimate the brightness of the ambient light. To achieve this, a preset ambient light adaptation map can be projected using the projection module. This preset ambient light adaptation map has high contrast and distinct bright and dark areas. An example of a preset ambient light adaptation map can be shown in Figure 11B.

[0496] S1103, the CMOS module takes a picture, obtains the eighth image, and stores it in the storage module.

[0497] S1104, the central processing module reads the eighth image from the storage module.

[0498] S1105, the central processing module estimates the current ambient brightness based on the eighth image.

[0499] For example, the central processing module 25 can calculate image features such as contrast and grayscale distribution of the eighth image; then, by comparing the image features of the eighth image with the image features of the source image corresponding to the eighth image, the brightness of the current environment can be estimated; and the current ambient brightness can be saved in the storage module 24.

[0500] S1106, the projection module adjusts the projection parameters according to the current ambient brightness.

[0501] For example, the projection module can adjust projection parameters based on the current ambient brightness; for example, projection parameters may include exposure, brightness, contrast, etc. For instance, if the current ambient brightness is low, the exposure or brightness of the projected image can be increased.

[0502] In other words, this application can realize the ambient light adaptive function based on CMOS components without relying on ALS; thus, it is not necessary to install ALS in the projection device, thereby reducing costs.

[0503] Automatic obstacle avoidance function

[0504] Figure 12 is a schematic diagram of an automatic obstacle avoidance process 1200 according to an embodiment of this application.

[0505] S1201, the control module switches the filter in the CMOS module corresponding to the lens position to an IR-CUT / all-pass filter and turns on the infrared fill light.

[0506] S1202, the projection module projects a preset white background image onto the projection medium.

[0507] S1203, the CMOS module takes a picture, obtains the ninth image, and stores it in the storage module.

[0508] S1204, the central processing module reads the ninth image from the storage module.

[0509] S1205, the central processing module detects the position and shape information of obstacles based on the ninth image.

[0510] For example, the central processing module 25 can use edge detection, contour detection or deep learning-based methods to detect obstacles in the projected image and save the position and shape information of the obstacles in the storage module 24.

[0511] S1206, the central processing module adjusts the position and shape of the source image based on the position and shape information of the obstacle.

[0512] For example, the source image can be scaled using the center point of the source image as a reference point until the projected image contains no obstacles.

[0513] Automatic screen entry function

[0514] Figure 13A is a schematic diagram of an automatic screen entry process 1300 according to an embodiment of this application.

[0515] S1301, the control module switches the filter in the CMOS module corresponding to the lens position to an IR-CUT / all-pass filter and turns on the infrared fill light.

[0516] S1302, the projection module projects a preset white background image onto the projection medium.

[0517] S1303, the CMOS module takes a picture, obtains the tenth image, and stores it in the storage module.

[0518] S1304, the central processing module reads the tenth image from the storage module.

[0519] S1305, the central processing module detects the position of the curtain based on the tenth image.

[0520] For example, the central processing module 25 can determine the coordinates of the four vertices of the screen by processing the tenth image. This step requires a preset white background image to surround the screen area. If the screen boundary is black, better detection results can be obtained. A typical screen, as shown in Figure 13B, has a clear boundary.

[0521] For example, the central processing module 25 can use corner detection algorithms (such as Harris corner detection) and edge detection algorithms (such as Canny and Sobel operators) to detect the tenth image and find clear edges; then, it can filter quadrilateral regions by rules such as area and angle to obtain the coordinates of the four vertices of the screen.

[0522] S1306, the central processing module adjusts the position and shape of the source image according to the position of the screen.

[0523] For example, the central processing module 25 can perform perspective transformation on the source image to transform the projected image into the screen area.

[0524] It should be understood that when the projection medium is not a screen or a rigid screen, the projection device 110 may not need to perform automatic screen entry processing.

[0525] In one example, FIG14 shows a schematic block diagram of an apparatus 1400 according to an embodiment of the present application. The apparatus 1400 may include a processor 1401 and a transceiver 1402, and optionally, a memory 1403, a projection lens assembly and a CMOS assembly.

[0526] The various components of device 1400 are coupled together via bus 1404, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 1404 in the figure.

[0527] Optionally, the memory 1403 can be used to store instructions from the foregoing method embodiments. The processor 1401 can be used to execute the instructions in the memory 1403, control the transceiver 1402 to receive signals, and control the transceiver 1402 to transmit signals.

[0528] Device 1400 may be a projection device or a chip of a projection device in the above method embodiments.

[0529] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0530] This application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above-described related method are executed to achieve the steps of the method in the above embodiments. The interface circuit is a transceiver 1402.

[0531] This embodiment also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on a projection device, the projection device performs the aforementioned method steps to implement the methods described in the above embodiments. Exemplarily, the computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0532] This embodiment also provides a computer program product containing computer instructions that, when executed by a computer or processor, cause the computer to perform the aforementioned steps to implement the methods described in the above embodiments. Exemplarily, the computer program product can be stored in random access memory (RAM), flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium well known in the art.

[0533] In this embodiment, the projection device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0534] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0535] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0536] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0537] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0538] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

An image processing apparatus, characterized in that, The image processing device includes a processing module and a complementary metal-oxide-semiconductor (CMOS) component. The CMOS component includes a CMOS module and an infrared fill light. The filter structure in the CMOS module allows infrared light to pass through. The CMOS module is used to capture images, including a first image captured towards the projection medium after the infrared fill light is turned on, and the projection medium displays the projected image. The processing module is configured to determine, based on the first image, whether the projected image is occluded by a target object; when it is determined that the projected image is occluded by the target object, it determines a first region of the projected image that is occluded and a second region in the source image that is occluded, wherein the second region is the region in the source image corresponding to the first region in the projected image. The apparatus according to claim 1, characterized in that, The CMOS module includes a camera, which includes a lens, a CMOS sensor, and one or more filter structures. The apparatus according to claim 1, characterized in that, The CMOS module includes multiple cameras, each of which includes a lens, a CMOS sensor, and a filter structure, and the filter structures in any two cameras are different. The apparatus according to claim 1, characterized in that, The CMOS module includes a first camera and a second camera. The first camera includes a lens, a CMOS sensor, and multiple filter structures. The second camera includes a lens, a CMOS sensor, and a filter structure. The apparatus according to any one of claims 1 to 4 is characterized in that, The filter structure includes an all-pass filter structure or a narrowband filter structure. The apparatus according to any one of claims 1 to 5 is characterized in that, The processing module is used for: Based on the first image, determine the region description information of the area where the target object is located in the first image; Obtain the region description information of the area where the projected image is located in the first image; Based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, it is determined whether the projected image is occluded by the target object. The apparatus according to claim 6, characterized in that, The processing module is also used for: Determine the difference between the region description information of the target object's location in the second image and the region description information of the target object's location in the third image, wherein the second image is the first image captured in this instance and the third image is the first image captured in the previous instance; When the difference information meets the first preset condition, based on the region description information of the area where the target object is located in the second image and the region description information of the area where the projected image is located in the second image, it is determined whether the projected image is occluded by the target object. The apparatus according to claim 7, characterized in that, The processing module is also used for: When the difference information does not meet the first preset condition, the motion speed information of the target object in the third image is obtained; Based on the motion speed information of the target object in the third image and the region description information of the area where the target object is located in the third image, the predicted region description information of the area of ​​the target object in the second image is determined. Based on the predicted region description information of the area of ​​the target object in the third image in the second image and the region description information of the area where the projected image is located in the second image, it is determined whether the projected image is occluded by the target object. The apparatus according to claim 6, characterized in that, The processing module is used for: The first image is processed using any one of a human detection algorithm, a human segmentation algorithm, or a depth estimation algorithm to obtain the region description information of the area where the target object is located in the first image; Based on the region description information of the area where the target object is located in the first image, determine whether the second preset condition is met; When the second preset condition is met, based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, it is determined whether the projected image is occluded by the target object. The apparatus according to claim 9 is characterized in that, The second preset condition includes that the area where the target object is located in the first image containing the target object does not overlap with the area where the projected image is located. The apparatus according to claim 6, characterized in that, The processing module is used for: Determine the similarity between each image block within a preset region in the first image and the corresponding image block within a preset region in the reference image; Based on the region description information of the connected region composed of target image blocks in the first image, the region description information of the region where the target object is located in the first image is determined, and the similarity between the target image block in the first image and the corresponding image block in the reference image is less than the similarity threshold. An image processing method, characterized in that, The method includes: Acquire a first image, which is taken by shooting towards a projection medium after the infrared fill light is turned on, and the projection medium displays a projected image. Based on the first image, determine whether the projected image is occluded by the target object; When it is determined that the projected image is occluded by the target object, a first region of the projected image that is occluded and a second region in the source image that is occluded are determined, wherein the second region is the region in the source image corresponding to the first region in the projected image. The method according to claim 12, characterized in that, The step of determining whether the projected image is occluded by the target object based on the first image includes: Based on the first image, determine the region description information of the area where the target object is located in the first image; Obtain the region description information of the area where the projected image is located in the first image; Based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, it is determined whether the projected image is occluded by the target object. The method according to claim 13, characterized in that, The method further includes: Determine the difference between the region description information of the target object's location in the second image and the region description information of the target object's location in the third image, wherein the second image is the first image acquired in this instance and the third image is the first image acquired in the previous instance; The step of determining whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image includes: When the difference information meets the first preset condition, based on the region description information of the area where the target object is located in the second image and the region description information of the area where the projected image is located in the second image, it is determined whether the projected image is occluded by the target object. The method according to claim 14, characterized in that, The step of determining whether the projected image is occluded by the target object based on the region description information of the region where the target object is located in the first image and the region description information of the region where the projected image is located in the first image further includes: When the difference information does not meet the first preset condition, the motion speed information of the target object in the third image is obtained; Based on the motion speed information of the target object in the third image and the region description information of the area where the target object is located in the third image, the predicted region description information of the area of ​​the target object in the second image is determined. Based on the predicted region description information of the area of ​​the target object in the third image in the second image and the region description information of the area where the projected image is located in the second image, it is determined whether the projected image is occluded by the target object. The method according to claim 13, characterized in that, The step of determining the region description information of the area where the target object is located in the first image based on the first image includes: The first image is processed using any one of a human detection algorithm, a human segmentation algorithm, or a depth estimation algorithm to obtain the region description information of the area where the target object is located in the first image; The step of determining whether the projected image is occluded by the target object based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image includes: Based on the region description information of the area where the target object is located in the first image, determine whether the second preset condition is met; When the second preset condition is met, based on the region description information of the area where the target object is located in the first image and the region description information of the area where the projected image is located in the first image, it is determined whether the projected image is occluded by the target object. The method according to claim 16, characterized in that, The second preset condition includes that the area where the target object is located in the first image containing the target object does not overlap with the area where the projected image is located. The method according to claim 13, characterized in that, The step of determining the region description information of the area where the target object is located in the first image based on the first image includes: Determine the similarity between each image block within a preset region in the first image and the corresponding image block within a preset region in the reference image; Based on the region description information of the connected region composed of target image blocks in the first image, the region description information of the region where the target object is located in the first image is determined, and the similarity between the target image block in the first image and the corresponding image block in the reference image is less than the similarity threshold. The method according to claim 16 or 17, characterized in that, The first image is processed using a depth estimation algorithm to obtain region description information of the area where the target object is located in the first image, including: The depth map of the first image is determined using the aforementioned depth estimation algorithm; The depth map of the image is binarized based on a binarization threshold to obtain a binarized image; Based on the region description information of the connected region formed by the target pixels in the binarized image, the region description information of the region where the target object is located is determined, and the pixel value of the target pixel in the binarized image is less than the binarization threshold. A projection device, characterized in that, The projection device includes a processing module, a complementary metal-oxide-semiconductor (CMOS) component, and a projection lens assembly. The CMOS component includes a CMOS module and an infrared fill light, and the filter component of the CMOS module allows infrared light to pass through. The projection lens assembly is used to project a first source image onto a projection medium to display a first projected image on the projection medium; The CMOS module is used to capture images, including a first image captured towards the projection medium after the infrared fill light is turned on; The processing module is used to determine, based on the first image, whether the first projected image is occluded by the target object; When it is determined that the first projected image is occluded by the target object, a first region of the first projected image that is occluded and a second region in the second source image that is occluded are determined, and the second region is the region in the second source image corresponding to the first region in the first projected image. The projection lens assembly is further configured to project a second source image of the obscured second region onto the projection medium to display a second projected image on the projection medium, wherein a third region in the second projected image is obscured, and the third region is the same as the first region. A projection device, characterized in that, The projection device includes: a memory, a processor, a complementary metal-oxide-semiconductor (CMOS) component, and a projection lens assembly. The CMOS component includes a CMOS module and an infrared fill light. The filter component of the CMOS module allows infrared light to pass through. The memory is coupled to the processor. The memory stores program instructions, which, when executed by the processor, cause the projection device to perform the following steps: Acquire a first image, which is captured by the CMOS component towards the projection medium after the infrared fill light is turned on, and the projection medium displays a projected image; Based on the first image, determine whether the projected image is occluded by the target object; When it is determined that the projected image is occluded by the target object, a first region of the projected image that is occluded and a second region in the source image that is occluded are determined, wherein the second region is the region in the source image corresponding to the first region in the projected image. A chip characterized in that, It includes one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, cause the steps of the method as described in any one of claims 12 to 19 to be performed. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 12 to 19. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a computer or processor, cause the steps of the method as described in any one of claims 12 to 19 to be performed.

Images