Image processing method, head-mounted display device, and computer-readable storage medium

Abstract

The present application relates to the technical field of visual positioning, and particularly relates to an image processing method, a head-mounted display device and a computer readable storage medium, wherein the method comprises the following steps: determining shadow feature information corresponding to at least two objects in an acquired image; calculating light field information in a current environment according to the shadow feature information, so as to determine a light source position in the current environment based on the light field information; and determining a relative position relationship between the objects according to the light source position and the shadow feature information. The relative position relationship between the objects in the environment and the objects can be determined even when there is no common reference object in the photographed image, and the problem of how to process the acquired image without recognizing the overlapping background or reference object is solved.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, and in particular to an image processing method, a head-mounted display device, and a computer-readable storage medium. Background Technology

[0002] In visual positioning solutions for AR (Virtual Reality) / VR (Virtual Reality) devices, images with the same background or reference objects are typically captured from different angles by cameras on the device to obtain the positions of objects around the device. Based on this relative positional relationship, visual positioning of VR / AR and other smart wearable devices is achieved.

[0003] However, due to the limited acquisition angle of the camera on the device, there may be situations where there is no overlapping background and / or reference between the captured images, resulting in the inability to obtain the position of objects around the device, thus causing positioning failure. Therefore, if images are acquired based solely on the same reference and / or background, visual positioning will be limited.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide an image processing method that addresses the problem of processing acquired images for which overlapping backgrounds or reference objects have not been identified.

[0006] To achieve the above objectives, the present invention provides an image processing method, the method comprising:

[0007] Determine the shadow feature information corresponding to at least two objects in the acquired image;

[0008] The light field information in the current environment is inferred based on the shadow feature information, so as to determine the position of the light source in the current environment based on the light field information;

[0009] The relative positional relationship between the objects is determined based on the position of the light source and the shadow feature information.

[0010] Optionally, the step of determining the shadow feature information corresponding to at least two objects in the acquired image includes:

[0011] Determine the grayscale value of each pixel in the image;

[0012] Target pixels whose gray values ​​fall within a preset gray range are identified as shadow pixels.

[0013] The target region in the image is extracted based on the shadow pixels and used as the shadow region.

[0014] Extract the feature information from the shadow region as the shadow feature information.

[0015] Optionally, there are multiple shadow regions, and before the step of extracting feature information from the shadow regions as the shadow feature information, the method further includes:

[0016] Identify each of the objects in the image and determine the object pixel coordinates of each of the objects; and,

[0017] Obtain the shadow pixel coordinates of each of the shadow regions;

[0018] Based on the relative positional relationship between the object pixel coordinates and the shadow pixel coordinates, the target shadow region corresponding to each object is matched;

[0019] The step of extracting feature information from the shadow region as the shadow feature information includes:

[0020] The feature information matching the target shadow region corresponding to the object is used as the shadow feature information of the object.

[0021] Optionally, the step of calculating the light field information in the current environment based on the shadow feature information includes:

[0022] The shadow length and shadow angle of the object, as well as the object height, are determined based on the shadow feature information.

[0023] The light field information is calculated based on the shadow length, the shadow angle, and the object height.

[0024] Optionally, the shadow feature information includes first shadow feature information and second shadow feature information, the image includes a first image and a second image, the object shadow in the first image corresponds to the first shadow feature information, the object shadow in the second image corresponds to the second shadow feature information, and before the step of determining the relative positional relationship between the objects based on the light source position and the shadow feature information, the method further includes:

[0025] The light field information is determined based on the first shadow feature information and / or the second shadow feature information, so as to determine the position of the light source based on the light field information;

[0026] The step of determining the relative positional relationship between the objects based on the light source position and the shadow feature information includes:

[0027] The relative positional relationship is determined based on the first shadow feature information, the second shadow feature information, and the position of the light source.

[0028] Optionally, before the step of calculating the light field information in the current environment based on the shadow feature information, and determining the position of the light source in the current environment based on the light field information, the method further includes:

[0029] Determine the number of shadows cast by the object in the image;

[0030] When there are multiple shadows, determine the light source type of the light source;

[0031] Based on the light source type, determine the target estimation strategy corresponding to the light source;

[0032] The step of calculating the light field information in the current environment based on the shadow feature information, and determining the position of the light source in the current environment based on the light field information, includes:

[0033] Based on the target estimation strategy, the light field information corresponding to each of the shadow feature information is estimated, so as to determine the position of the light source based on the light field information.

[0034] Optionally, the light source type includes point light sources and parallel light sources, and the step of determining the light source type of each light source includes:

[0035] Determine the contrast of each of the shadows;

[0036] When the contrast is less than or equal to a preset contrast threshold, the light source corresponding to the shadow is determined to be the parallel light source;

[0037] When the contrast is greater than a preset contrast threshold, the light source corresponding to the shadow is determined to be the point light source;

[0038] And / or, determine the shadow direction of each object in the image;

[0039] When the shadows are in the same direction, the light source corresponding to the shadows in the same direction is determined to be a parallel light source;

[0040] When the shadow direction is opposite, the light source corresponding to the opposite shadow is determined to be a point light source.

[0041] Optionally, the step of determining the light field estimation strategy corresponding to the light source based on the light source type includes:

[0042] When the light source type is the point light source, obtain the gray value of the shadow corresponding to the point light source;

[0043] The light intensity of the point light source is calculated based on the grayscale value;

[0044] A parallel light source that satisfies the light source intensity is identified, and the light field information corresponding to each shadow feature information is calculated based on the light field estimation strategy corresponding to the parallel light source.

[0045] Optionally, the shadow feature information includes shadow contrast, and the step of calculating the light field information of the current environment based on the shadow feature information includes:

[0046] The relative distance between the light source and the object is determined based on the shadow contrast.

[0047] The light field information is calculated based on the relative distance, and the position of the light source in the current environment is determined based on the light field information.

[0048] Optionally, before the step of determining the shadow feature information corresponding to at least two objects in the acquired image, the method further includes:

[0049] Determine whether the object's shadow exists in the image;

[0050] If it does not exist, the light field information corresponding to the environment in which the object is located is determined as the preset first light field information;

[0051] If present, proceed with the step of determining the shadow feature information corresponding to at least two objects in the acquired image.

[0052] Optionally, after determining the relative positional relationship between the objects based on the position of the light source, the method further includes:

[0053] Obtain its own global location information on the global map;

[0054] Based on the global positioning information and the relative positional relationship, an indoor positioning operation is performed.

[0055] Optionally, before the step of determining the shadow feature information corresponding to at least two objects in the acquired image, the method further includes:

[0056] Images of the target object are acquired at preset intervals;

[0057] Determine the amount of shadow change of the target object in the current image captured in the current cycle compared to the historical image captured in the previous cycle, as well as the amount of position change of the target object;

[0058] Calculate the trend of light and object changes in the current environment based on the shadow change and the position change.

[0059] The step of determining the shadow feature information corresponding to at least two objects in the acquired image includes:

[0060] Based on the trend of light changes, determine the shadow feature information of at least two objects in the acquired image.

[0061] In addition, to achieve the above objectives, the present invention also provides a head-mounted display device, the head-mounted display device comprising: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the steps of the image processing method as described above.

[0062] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0063] This invention provides an image processing method, a head-mounted display device, and a computer-readable storage medium. After determining the shadow feature information corresponding to at least two objects in the acquired image, the light field information of the current environment is determined based on the shadow feature information, and the position of the light source is determined based on the light field information. Finally, the relative positional relationship between objects is determined based on the position of the light source and the shadow feature information, thereby determining the relative positional relationship between objects in the environment when there are no common reference objects in the captured image. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the hardware operating environment of the head-mounted display device according to an embodiment of the present invention;

[0065] Figure 2 This is a scene diagram of the image processing method of the present invention;

[0066] Figure 3 This is a flowchart illustrating the first embodiment of the image processing method of the present invention;

[0067] Figure 4 This is a flowchart illustrating a second embodiment of the image processing method of the present invention;

[0068] Figure 5 This is a schematic diagram of a composite light source scene in the image processing method of the present invention;

[0069] Figure 6 This is a flowchart illustrating the third embodiment of the image processing method of the present invention;

[0070] Figure 7 This is a flowchart illustrating the fourth embodiment of the image processing method of the present invention;

[0071] Figure 8 This is a flowchart illustrating the fifth embodiment of the image processing method of the present invention.

[0072] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0073] This application calculates the light field information of the current environment by extracting the shadow features of objects in the acquired image, then predicts the position of the light source in the environment based on the light field information, and then determines the relative position between various objects in the environment based on the position of the light source, thereby realizing the positioning of objects in the environment based on the relative position between objects.

[0074] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0075] As one implementation scheme, Figure 1 This is a schematic diagram of the hardware operating environment of the head-mounted display device involved in the embodiments of the present invention.

[0076] like Figure 1 As shown, the head-mounted display device may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0077] Those skilled in the art will understand that Figure 1 The architecture of the head-mounted display device shown in the figure does not constitute a limitation on the head-mounted display device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0078] like Figure 1As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an image processing program. The operating system is a program that manages and controls the hardware and software resources of the head-mounted display device, as well as the image processing program and the operation of other software or programs.

[0079] exist Figure 1 In the architecture of the head-mounted display device shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used to connect to the back-end server and communicate data with the back-end server; the processor 1001 can be used to call the image processing program stored in the memory 1005.

[0080] In this embodiment, the head-mounted display device includes: a memory 1005, a processor 1001, and an image processing program stored in the memory and executable on the processor, wherein:

[0081] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0082] Determine the shadow feature information corresponding to at least two objects in the acquired image;

[0083] The light field information in the current environment is inferred based on the shadow feature information, so as to determine the position of the light source in the current environment based on the light field information;

[0084] The relative positional relationship between the objects is determined based on the position of the light source and the shadow feature information.

[0085] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0086] Determine the grayscale value of each pixel in the image;

[0087] Target pixels whose gray values ​​fall within a preset gray range are identified as shadow pixels.

[0088] The target region in the image is extracted based on the shadow pixels and used as the shadow region.

[0089] Extract the feature information from the shadow region as the shadow feature information.

[0090] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0091] Identify each of the objects in the image and determine the object pixel coordinates of each of the objects; and,

[0092] Obtain the shadow pixel coordinates of each of the shadow regions;

[0093] Based on the relative positional relationship between the object pixel coordinates and the shadow pixel coordinates, the target shadow region corresponding to each object is matched;

[0094] The feature information matching the target shadow region corresponding to the object is used as the shadow feature information of the object.

[0095] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0096] The shadow length and shadow angle of the object, as well as the object height, are determined based on the shadow feature information.

[0097] The light field information is calculated based on the shadow length, the shadow angle, and the object height.

[0098] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0099] The light field information is determined based on the first shadow feature information and / or the second shadow feature information, so as to determine the position of the light source based on the light field information;

[0100] The relative positional relationship is determined based on the first shadow feature information, the second shadow feature information, and the position of the light source.

[0101] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0102] Determine the number of shadows cast by the object in the image;

[0103] When there are multiple shadows, determine the light source type of the light source;

[0104] Based on the light source type, determine the target estimation strategy corresponding to the light source;

[0105] Based on the target estimation strategy, the light field information corresponding to each of the shadow feature information is estimated, so as to determine the position of the light source based on the light field information.

[0106] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0107] Determine the contrast of each of the shadows;

[0108] When the contrast is less than or equal to a preset contrast threshold, the light source corresponding to the shadow is determined to be the parallel light source;

[0109] When the contrast is greater than a preset contrast threshold, it is determined that the light source corresponding to the shadow is the point light source;

[0110] And / or, determine the shadow direction of each object in the image;

[0111] When the shadows are in the same direction, the light source corresponding to the shadows in the same direction is determined to be a parallel light source;

[0112] When the shadow direction is opposite, the light source corresponding to the opposite shadow is determined to be a point light source.

[0113] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0114] When the light source type is the point light source, obtain the gray value of the shadow corresponding to the point light source;

[0115] The light intensity of the point light source is calculated based on the grayscale value;

[0116] A parallel light source that satisfies the light source intensity is identified, and the light field information corresponding to each shadow feature information is calculated based on the light field estimation strategy corresponding to the parallel light source.

[0117] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0118] The relative distance between the light source and the object is determined based on the shadow contrast.

[0119] The light field information is calculated based on the relative distance, and the position of the light source in the current environment is determined based on the light field information.

[0120] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0121] Determine whether the object's shadow exists in the image;

[0122] If it does not exist, the light field information corresponding to the environment in which the object is located is determined as the preset first light field information;

[0123] If present, proceed with the step of determining the shadow feature information corresponding to at least two objects in the acquired image.

[0124] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0125] Obtain its own global location information on the global map;

[0126] Based on the global positioning information and the relative positional relationship, an indoor positioning operation is performed.

[0127] When processor 1001 calls the image processing program stored in memory 1005, it performs the following operations:

[0128] Images of the target object are acquired at preset intervals;

[0129] Determine the amount of shadow change of the target object in the current image captured in the current cycle compared to the historical image captured in the previous cycle, as well as the amount of position change of the target object;

[0130] Calculate the trend of light and object changes in the current environment based on the shadow change and the position change.

[0131] The step of determining the shadow feature information corresponding to at least two objects in the acquired image includes:

[0132] Based on the trend of light changes, determine the shadow feature information of at least two objects in the acquired image.

[0133] Based on the hardware architecture of the head-mounted display device based on the above-mentioned visual positioning technology, an embodiment of the image processing method of the present invention is proposed.

[0134] In some practical applications, due to the limited shooting angle of head-mounted display cameras, the captured images may lack a common background or reference object, making it impossible to analyze the relative positional relationships between objects. Figure 2 The room has a point light source and four fixed objects. The three objects on the left can be captured simultaneously, but the objects on the right cannot be captured. After the camera captures A and B respectively, since there is no overlapping background or reference between A and B, the reference-based image processing algorithm cannot be used.

[0135] However, through Figure 2 It can be seen that although the four objects are located in different positions in the room, they are all under the same point light source, and their shadows are all produced by the same point light source. Therefore, by using the characteristic information of the shadows of these four objects in different positions, the position of the light source in the room can be deduced. After knowing the position of the light source, the light source can be used as a reference to determine the relative position between the four objects. Thus, indoor positioning can be achieved when there is no reference between the captured images.

[0136] Reference Figure 3 In the first embodiment, the image processing method includes the following steps:

[0137] Step S10: Determine the shadow feature information corresponding to at least two objects in the acquired image;

[0138] In this embodiment, the method is applied to a head-mounted display device equipped with a camera. This camera captures images of the environment in which the head-mounted display device is currently located, and the images contain at least two objects and their corresponding shadows. In this step, shadow feature extraction processing is performed on the captured images to determine the shadow feature information of the objects in the image.

[0139] It should be noted that in this embodiment, the acquisition method involves multiple acquisitions, each from a different angle, and each acquired image contains at least one object and its corresponding shadow, with no identical reference objects between the images. For example, if there are two objects in the current environment, and the acquired images (here referred to as the first image and the second image) have no identical reference objects, then the first image should contain one object and its corresponding shadow, while the second image should contain the other object and its corresponding shadow.

[0140] Furthermore, it is understandable that if at least two of the multiple images captured at this time contain two objects simultaneously, it means that the captured images have a common reference object. In this case, the image processing algorithm based on the reference object is called to determine the positional relationship between the objects, and this step is unnecessary.

[0141] Optionally, regarding how to determine shadow features in an image, since the grayscale values ​​of shadow areas are darker than other areas, shadow features can be determined using the grayscale values ​​of pixels in the image. Specifically, the acquired image is first processed into a grayscale image, and then the grayscale value of each pixel in the image is determined. Then, target pixels with grayscale values ​​within a preset grayscale range are identified as shadow pixels (the grayscale range can be obtained by training a machine vision model with a large number of object shadow images, or by operators adjusting parameters). The target area formed by these shadow pixels is then considered the shadow area. Finally, feature information from the shadow area is extracted as shadow feature information.

[0142] Optionally, when multiple objects exist in the image, multiple shadow regions and shadow feature information need to be extracted. In this scenario, it is also necessary to match the shadow region with its connected objects to avoid discrepancies between the located light field information and the actual light field information in the environment due to the shadow region being identified as the shadow of another unrelated object. Specifically, firstly, each object in the image is identified, and the corresponding pixel coordinates of each object are determined as object pixel coordinates. Then, the pixel coordinates of each shadow region are obtained as shadow pixel coordinates. Next, since the shadows of objects in the image are usually connected to the objects, the object pixel coordinates that are closest in relative position to the shadow pixel coordinates are selected as the target shadow region for matching the object. The characteristic information of this target shadow region is extracted as the shadow feature information of the object.

[0143] Optionally, the shadow feature information includes information such as the length, angle, and depth of the shadow, and may also include pixel parameters of the shadow area, such as grayscale value and contrast.

[0144] Step S20: Calculate the light field information in the current environment based on the shadow feature information, so as to determine the position of the light source in the current environment based on the light field information;

[0145] In this embodiment, after determining the shadow feature information, the light field information of the current environment is inferred based on the shadow feature information. The light field information is characterized by information such as the light intensity, light source position, and dispersion direction in the current environment.

[0146] Optionally, based on the shadow feature information, the angle θ, length l, and object height h of the object corresponding to the shadow are determined. Then, based on θ, l, and h, the light field function H(θ, l, h) of the current environment is calculated using geometric optics. The specific method of calculating the light field based on shadow is relatively existing and will not be explained in detail here.

[0147] Optionally, since the closer the distance between the light source and the object, the deeper the shadow, the relative distance between the light source and the object can be calculated based on the shadow contrast. The light field information can then be calculated based on the relative distance, and finally the position of the light source in the current environment can be determined based on the light field information.

[0148] After determining the light field information and obtaining the light field function, the position of the light source can be determined from the light field information. The position of the light source is characterized as the absolute position of the light source in the current three-dimensional space of the environment. It does not change with the movement of objects or head-mounted display devices. Therefore, it can be used as a reference point to confirm the position of other objects.

[0149] Step S30: Determine the relative positional relationship between the objects based on the light source position and the shadow feature information;

[0150] In this embodiment, after determining the position of the light source, the relative positional relationship between objects in the image is determined based on the position of the light source and the shadow feature information.

[0151] Optionally, at least two images need to be acquired for determination. In some implementations, two images are acquired (hereinafter referred to as the first image and the second image). The object shadow in the first image corresponds to the first shadow feature information, and the object shadow in the second image corresponds to the second shadow feature information. Specifically, the light field information is determined based on one or both of the first and second shadow feature information, thereby determining the position of the light source. It should be noted that the more shadow feature information there is, the more accurate the determined light field information will be.

[0152] After determining the position of the light source, the relative position relationship between the light source and the two objects in different images is determined based on the position of the light source, the first shadow feature information, and the second shadow feature information.

[0153] Optionally, after determining the relative positional relationship between objects, the head-mounted display device obtains its own global positioning information in the global map, and performs indoor positioning operations based on the global positioning information and the relative positional relationship.

[0154] In the technical solution provided in this embodiment, after determining the shadow feature information corresponding to at least two objects in the acquired image, the light field information in the current environment is determined based on the shadow feature information. The position of the light source is determined based on the light field information. Finally, the relative positional relationship between objects is determined based on the position of the light source and the shadow feature information. Thus, the relative positional relationship between objects in the environment is determined when there is no common reference in the captured image.

[0155] Reference Figure 4 In the second embodiment, based on the first embodiment, before step S20, the method further includes:

[0156] Step S40: Determine the number of shadows cast by the object in the image;

[0157] Step S50: When there are multiple shadows, determine the light source type of the light source;

[0158] Step S60: Based on the light source type, determine the target estimation strategy corresponding to the light source;

[0159] Step S20 includes:

[0160] Step S21: Based on the target estimation strategy, calculate the light field information corresponding to each of the shadow feature information, so as to determine the position of the light source based on the light field information.

[0161] Optionally, in this embodiment, when the device is located in an indoor environment, the indoor light field is often a composite light source environment composed of multiple types of light sources superimposed, such as a room with windows, a multi-spotlight lighting system, and complex light strips. For example, such as Figure 5 In the complex lighting environment shown, when sunlight (parallel light source) shines into the room through the window, the shadow of the same object will change compared to electric lighting (point light source). The object will simultaneously have both point light source shadow and parallel light source shadow. In this scenario, more information about the object's shadow is needed to determine the position of the light source.

[0162] Specifically, in this embodiment, the number of shadows of the object in the image is first determined. The purpose of this is to determine whether there is light source superposition in the indoor environment. If the object has multiple (more than two) shadows, it is determined that there is light source superposition.

[0163] Furthermore, when there is a superposition of light sources, it is necessary to determine the type of light source. In this embodiment, the light source type is mainly point light source type and parallel light source type. Point light source is characterized by indoor lighting, and parallel light source is characterized by outdoor light (such as sunlight) shining into the room.

[0164] Optionally, since the point light source is closer to the object and its intensity is stronger than that of the parallel light source, the contrast of the shadow of the point light source is higher than that of the shadow of the parallel light source. Therefore, the type of point light source can be determined based on the contrast of the shadow. When the contrast is greater than a preset contrast threshold, the light source is determined to be a point light source. When the contrast is less than or equal to the contrast threshold, it is a parallel light source.

[0165] Alternatively, the type of light source can be determined based on the direction of the shadows. Under a parallel light source, the shadows of different objects are parallel and of the same length, while under a point light source, the angles and lengths of the shadows of different objects are not the same. Therefore, when the shadows of different objects in an image are in the same direction, the light source corresponding to these shadows is a parallel light source; if they are in opposite directions, it is a point light source.

[0166] Once the different light source types are identified, the target estimation strategy corresponding to that light source type is selected to calculate the light field information, thereby determining the location of the light source.

[0167] In the technical solution provided in this embodiment, when there are multiple shadows, the shadow type of each shadow is determined, and then different types of strategies are called to calculate the light field information, thereby determining the position of the light source.

[0168] Reference Figure 6 In the third embodiment, based on any embodiment, step S60 includes:

[0169] Step S61: When the light source type is the point light source, obtain the gray value of the shadow corresponding to the point light source;

[0170] Step S62: Calculate the light source intensity of the point light source based on the gray value;

[0171] Step S63: Determine the parallel light source that satisfies the light source intensity, and calculate the light field information corresponding to each shadow feature based on the light field estimation strategy corresponding to the parallel light source.

[0172] Optionally, in this embodiment, when the light source type is a point light source, since the calculation of a point light source is relatively complex, to facilitate the calculation, an equivalent parallel light position applicable to different room sizes will be set, or an equidistant parallel light source will be calculated based on the position of the point light source. Specifically, the grayscale value of the shadow corresponding to the point light source is obtained. The grayscale value is proportional to the light source intensity. Therefore, after calculating the light source intensity of the point light source based on the grayscale value, a parallel light source that meets the light source intensity value is determined to replace the point light source, and the light field estimation strategy corresponding to the point light source is selected to estimate the light field information.

[0173] In the technical solution provided in this embodiment, the computational difficulty is reduced by converting a point light source into a parallel light source at equal distances.

[0174] Reference Figure 7 In the fourth embodiment, based on any embodiment, before step S10, the following step is further included:

[0175] Step S70: Determine whether the shadow of the object exists in the image;

[0176] Step S81: If it does not exist, determine the light field information corresponding to the environment in which the object is located as the preset first light field information;

[0177] Step S82: If present, perform the step of determining the shadow feature information corresponding to at least two objects in the acquired image.

[0178] Optionally, in this embodiment, before extracting shadows from the image, it is necessary to determine whether the object's shadow exists in the image. If the acquired image does not contain the object's shadow, it indicates that the light intensity in all directions at the object's current position is consistent, resulting in no shadow on the object, or that the object is under shadowless lighting. In this scenario, the object's light field information corresponds to a preset first light field information, where the light field parameter values ​​in the first light field information are all preset fixed values, and the determined first light source position is also a fixed position. Optionally, the first light source position is directly above the object, or four equally spaced light sources are set in the four directions E (East), S (South), W (West), and N (North) at a 45° vertical angle to the object.

[0179] If the acquired image contains the shadow of an object, then step S10 in the first embodiment is executed.

[0180] In the technical solution provided in this embodiment, if there is no shadow in the image, the light field information of the environment in which the object is located is determined to a preset value, so that the position of the light source can be determined even without shadows.

[0181] Reference Figure 8 In the fifth embodiment, based on any embodiment, before step S10, the following step is further included:

[0182] Step S90: Acquire images of the target object at preset intervals;

[0183] Step S100: Determine the amount of shadow change of the target object in the current image captured in the current cycle compared to the historical image captured in the previous cycle, as well as the amount of position change of the target object.

[0184] Step S110: Calculate the trend of light and object changes in the current environment based on the shadow change and the position change.

[0185] The step of determining the shadow feature information corresponding to at least two objects in the acquired image includes:

[0186] Step S11: Based on the trend of light changes, determine the shadow feature information corresponding to at least two objects in the acquired image.

[0187] Optionally, since the positions of objects and shadows may change over time, this embodiment introduces a time variable (i.e., the trend of light and object change). The trend of light and object change in the current environment is determined by acquiring shadow images over a continuous period, and then shadow feature information is determined based on this trend. The trend of light and object change is characterized as the trend of relative positional change between the light source and the object over a period of time.

[0188] Specifically, images of the target object are first acquired at preset intervals, optionally with an interval of 0.2 seconds. The changes in position and shadow of the same object (the target object) in the current image captured in the current interval and in historical images captured in the previous interval are determined. The shadow change includes the shadow length and angle (direction), while the position change is used to determine the object's rate of change. Next, based on the shadow and position changes obtained over multiple intervals, the trend of light and object changes in the current environment is determined.

[0189] In the technical solution provided in this embodiment, it is considered that the positions of objects and shadows may change. By introducing a time variable to extract shadow features, the accuracy of feature extraction is improved, thereby improving the accuracy of light source position positioning.

[0190] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the head-mounted display device to implement the process steps of the embodiments of the above methods.

[0191] Therefore, the present invention also provides a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the various steps of the image processing method described in the above embodiments.

[0192] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0193] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0194] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0196] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0197] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0198] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, third, etc., does not indicate any order. These words can be interpreted as names.

[0199] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0200] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An image processing method, characterized in that, The method includes the following steps: The shadow feature information corresponding to at least two objects in the acquired image is determined. The shadow feature information includes first shadow feature information and second shadow feature information. The image includes a first image and a second image. The shadow of an object in the first image corresponds to the first shadow feature information, and the shadow of an object in the second image corresponds to the second shadow feature information. The shadow feature information includes shadow contrast. Based on the first shadow feature information and / or the second shadow feature information, light field information is determined to determine the position of the light source, including determining the relative distance between the light source and the object based on the shadow contrast; the light field information is calculated based on the relative distance to determine the position of the light source in the current environment, wherein the light field information represents the light intensity, position of the light source, and dispersion direction in the current environment, and the position of the light source represents the absolute position of the light source in the three-dimensional space of the current environment; Determining the relative positional relationship between the objects based on the light source position and the shadow feature information includes determining the relative positional relationship based on the first shadow feature information, the second shadow feature information, and the light source position.

2. The method as described in claim 1, characterized in that, The step of determining the shadow feature information corresponding to at least two objects in the acquired image includes: Determine the grayscale value of each pixel in the image; Target pixels whose gray values ​​fall within a preset gray range are identified as shadow pixels. The target region in the image is extracted based on the shadow pixels and used as the shadow region. Extract the feature information from the shadow region as the shadow feature information.

3. The method as described in claim 2, characterized in that, The shadow regions are multiple, and before the step of extracting feature information from the shadow regions as the shadow feature information, the method further includes: Identify each of the objects in the image and determine the object pixel coordinates of each of the objects; and, Obtain the shadow pixel coordinates of each of the shadow regions; Based on the relative positional relationship between the object pixel coordinates and the shadow pixel coordinates, the target shadow region corresponding to each object is matched; The step of extracting feature information from the shadow region as the shadow feature information includes: The feature information matching the target shadow region corresponding to the object is used as the shadow feature information of the object.

4. The method as described in claim 1, characterized in that, The step of determining the light field information based on the first shadow feature information and / or the second shadow feature information, and then determining the position of the light source based on the light field information, includes: The shadow length and shadow angle of the object, as well as the object height, are determined based on the shadow feature information. The light field information is calculated based on the shadow length, the shadow angle, and the object height.

5. The method as described in claim 1, characterized in that, After the step of determining the shadow feature information corresponding to at least two objects in the acquired image, the method further includes: Determine the number of shadows cast by the object in the image; When there are multiple shadows, determine the light source type of the light source; Based on the light source type, determine the target estimation strategy corresponding to the light source; Based on the target estimation strategy, the light field information corresponding to each of the shadow feature information is estimated, so as to determine the position of the light source based on the light field information.

6. The method as described in claim 5, characterized in that, The light source type includes point light sources and parallel light sources, and the step of determining the light source type includes: Determine the contrast of each of the shadows; When the contrast is less than or equal to a preset contrast threshold, the light source corresponding to the shadow is determined to be the parallel light source; When the contrast is greater than a preset contrast threshold, the light source corresponding to the shadow is determined to be the point light source; And / or, determine the shadow direction of each object in the image; When the shadows are in the same direction, the light source corresponding to the shadows in the same direction is determined to be a parallel light source; When the shadow direction is opposite, the light source corresponding to the opposite shadow is determined to be a point light source.

7. The method as described in claim 6, characterized in that, The step of determining the target estimation strategy corresponding to the light source based on the light source type includes: When the light source type is the point light source, obtain the gray value of the shadow corresponding to the point light source; The light intensity of the point light source is calculated based on the grayscale value; A parallel light source that satisfies the light source intensity is identified, and the light field information corresponding to each shadow feature information is calculated based on the light field estimation strategy corresponding to the parallel light source.

8. The method as described in claim 1, characterized in that, Before the step of determining the shadow feature information corresponding to at least two objects in the acquired image, the method further includes: Determine whether the object's shadow exists in the image; If it does not exist, the light field information corresponding to the environment in which the object is located is determined as the preset first light field information; If present, proceed with the step of determining the shadow feature information corresponding to at least two objects in the acquired image.

9. The method as described in claim 1, characterized in that, After the step of determining the relative positional relationship between the objects based on the light source position and the shadow feature information, the method further includes: Obtain its own global location information on the global map; Based on the global positioning information and the relative positional relationship, an indoor positioning operation is performed.

10. The method as described in claim 1, characterized in that, Before the step of determining the shadow feature information corresponding to at least two objects in the acquired image, the method further includes: Images of the target object are acquired at preset intervals; Determine the amount of shadow change of the target object in the current image captured in the current cycle compared to the historical image captured in the previous cycle, as well as the amount of position change of the target object; Calculate the trend of light and object changes in the current environment based on the shadow change and the position change. The step of determining the shadow feature information corresponding to at least two objects in the acquired image includes: Based on the trend of light changes, determine the shadow feature information of at least two objects in the acquired image.

11. A head-mounted display device, characterized in that, The head-mounted display device includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the steps of the image processing method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image processing program, which, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 10.

Images