Image processing method, image processing device, augmented reality device, and medium

Abstract

This application provides an image processing method, an image processing apparatus, an augmented reality device, and a medium. The method is applied to an augmented reality device, which includes a laser projector and a camera. The image processing method includes: responding to an image acquisition command, controlling the camera to acquire an image of a target object containing a light spot, wherein the light spot is a bright spot formed by a laser beam emitted by the laser projector projected onto the surface of the target object; determining the pixel coordinates of the light spot in the target object image based on the mapping relationship between the output axis of the laser projector and the imaging optical axis of the camera; acquiring a local image from the target object image based on the pixel coordinates and performing recognition processing on the local image; and displaying the recognition result. This method can quickly locate the light spot and perform recognition processing only on a local image, reducing the difficulty of framing and alignment caused by the deviation between the camera's imaging optical axis and the visual center, reducing computational overhead, and thereby improving the stability and reliability of the recognition result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, an image processing apparatus, an augmented reality device, and a medium. Background Technology

[0002] In recent years, with the development of Augmented Reality (AR) technology, AR glasses have been widely used in application scenarios such as document scanning, long-distance target recognition, and information assistance prompts because they can integrate virtual information with real scenes.

[0003] However, taking head-mounted augmented reality glasses as an example, they typically rely on an integrated camera to capture full-frame images of the real-world environment and perform target recognition or text recognition processing based on the captured image data. Because there is a deviation between the camera's imaging optical axis and the user's visual center, it is difficult for the user to determine the alignment between the camera's viewfinder and the target object. In actual use, users often need to repeatedly turn their heads or adjust their posture to complete the framing, thus reducing interaction efficiency. Furthermore, it is difficult to balance computational efficiency with the stability and adaptability requirements of augmented reality devices in complex environments. Summary of the Invention

[0004] The image processing method, image processing apparatus, augmented reality device, and medium provided in the embodiments of this application aim to solve at least some of the defects existing in the image processing methods of augmented reality devices.

[0005] In a first aspect, embodiments of this application provide an image processing method. This image processing method is applied to an augmented reality device, which includes a laser projector and a camera. The method includes: responding to an image acquisition command, controlling the camera to acquire an image of a target object containing a light spot, wherein the light spot is a bright spot formed by a laser beam emitted by the laser projector projected onto the surface of the target object; determining the pixel coordinates of the light spot in the target object image based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera; acquiring a local image from the target object image based on the pixel coordinates, and performing recognition processing on the local image; and displaying the recognition result.

[0006] In some embodiments, controlling the camera to acquire an image of a target object containing a light spot in response to an image acquisition command includes: detecting a change in the posture of the augmented reality device; determining that the laser beam emitted by the laser projector is pointing towards the target object when the change in posture meets a preset pointing condition; and controlling the camera to acquire an image of the target object containing a light spot when a trigger signal is detected; wherein the preset pointing condition is used to indicate a determination condition that the emission direction of the laser projector is substantially consistent with the direction of the target object.

[0007] In some embodiments, determining the pixel coordinates of the light spot in the target object image based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera includes: acquiring the intrinsic parameters of the laser projector; determining the predicted projection position of the emission axis of the laser projector on the imaging plane of the camera based on the intrinsic parameters and the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera; determining the candidate pixel position of the light spot based on the predicted projection position and the target object image; and determining the pixel coordinates of the light spot in the target object image based on the candidate pixel position; wherein the intrinsic parameters include at least focal length parameters and offset angle parameters.

[0008] In some embodiments, determining the candidate pixel position of the light spot based on the predicted projection position and the target object image includes: taking the predicted projection position as the center point, cropping a first local image of a first preset size from the target object image; performing brightness analysis on each pixel in the first local image and filtering out pixels with brightness higher than a preset brightness threshold; determining at least one brightness concentration region based on the position information of the filtered pixels; calculating the center position of each brightness concentration region; and determining the center position as the candidate pixel position of the light spot.

[0009] In some embodiments, determining the pixel coordinates of the light spot in the target object image based on the candidate pixel positions includes: when there are multiple candidate pixel positions in the target object image, determining the distance between each candidate pixel position and the predicted projection position; selecting the candidate pixel position with the smallest distance to the predicted projection position from the multiple candidate pixel positions as the target pixel position; and using the pixel coordinates corresponding to the target pixel position as the pixel coordinates of the light spot in the target object image.

[0010] In some embodiments, the step of obtaining a local image from the target object image based on the pixel coordinates and performing recognition processing on the local image includes: cropping a second local image of a second preset size from the target object image with the pixel coordinates of the light spot in the target object image as the center point; performing resolution enhancement processing on the second local image to obtain an enhanced image; and performing recognition processing on the enhanced image to obtain recognition information in the enhanced image; wherein the recognition information includes at least one of text information, image content information, and target object information.

[0011] In some embodiments, the display recognition result includes: determining the projection display position of the laser beam emitted by the laser projector on a preset projection plane based on the principle of rectilinear propagation of light; mapping the projection display position to the near-eye display coordinate system of the augmented reality device according to a preset mapping relationship to obtain the corresponding target display position; and displaying the magnified augmented image and the recognition information at the target display position.

[0012] Secondly, embodiments of this application provide an image processing apparatus. The image processing apparatus includes: a control module, configured to control a camera to acquire an image of a target object containing a light spot in response to an image acquisition command, wherein the light spot is a bright spot formed by a laser beam emitted by a laser projector projected onto the surface of the target object; a determination module, configured to determine the pixel coordinates of the light spot in the target object image based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera; an identification module, configured to acquire a local image from the target object image based on the pixel coordinates and perform identification processing on the local image; and a display module, configured to display the identification result.

[0013] Thirdly, embodiments of this application provide an augmented reality device. The augmented reality device includes: a laser projector for emitting a laser beam toward a target object to form a light spot on the surface of the target object; a camera for acquiring an image of the target object containing the light spot; a processor communicatively connected to the laser projector and the camera; and a memory communicatively connected to the processor, the memory storing computer program instructions, which, when invoked by the processor, cause the processor to execute the image processing method described above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing processor-executable computer program instructions, which, when executed by the processor, cause the computer to perform the image processing method described above.

[0015] The beneficial effects of the image processing method provided in this application are as follows: By utilizing the mapping relationship between the output axis of the laser projector and the imaging optical axis of the camera, and using the light spot formed by the laser beam on the surface of the target object as a physical indicator, the position of the light spot can be quickly determined in the target object image captured by the camera when an image acquisition command is detected. This reduces the difficulty of framing and alignment caused by the deviation between the camera's imaging optical axis and the visual center, reduces the interaction cost caused by repeated posture adjustments, and improves the convenience and efficiency of information acquisition. In addition, this method only performs recognition processing on the local image corresponding to the light spot, without needing to continuously perform high-computation processing such as global feature extraction and sliding window detection on the entire image. This reduces the number of pixels involved in the calculation and the number of operations, thereby reducing processor load and power consumption. At the same time, it can maintain the stability and reliability of the recognition results under complex lighting changes or background interference conditions, which is beneficial to the long-term stable operation of augmented reality devices. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0017] Figure 1 A schematic diagram illustrating the image processing method provided in an embodiment of this application; Figure 2 A schematic diagram of a sub-process of step S110 of the image processing method provided in an embodiment of this application; Figure 3 A schematic diagram of a sub-process of step S120 of the image processing method provided in the embodiments of this application; Figure 4 A schematic diagram of a sub-process of step S123 of the image processing method provided in the embodiments of this application; Figure 5 A schematic diagram of a sub-process of step S124 of the image processing method provided in the embodiments of this application; Figure 6 A schematic diagram of a sub-process of step S130 of the image processing method provided in an embodiment of this application; Figure 7 A schematic diagram of a sub-process of step S140 of the image processing method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the architecture of the image processing apparatus provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of an augmented reality device provided in an embodiment of this application. Detailed Implementation

[0018] To facilitate understanding of this application, a more detailed description of this application will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0019] It should be noted that when a component is said to be "set on" another component, it can be directly on the other component or there may be an intervening component. When a component is considered to be "connected" to another component, it can be directly connected to the other component or there may be an intervening component, or it can refer to the two components being interconnected via signals. When a component is considered to be "coupled" to another component, it can be directly coupled to the other component or there may be an intervening component, or it can refer to the two components interacting via signals.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0021] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0022] In practical applications of augmented reality (AR) devices, it is typically necessary to utilize the integrated camera to capture and recognize images of target objects in the real-world environment to achieve functions such as document scanning, long-distance information reading, and assisted travel guidance. Typical AR image processing methods include: First, the camera captures images of the real-world scene within the user's field of view to obtain a full-frame image containing the target object; Then, target detection, text recognition and other processing are performed on the entire acquired image to obtain text or graphic information in the target object; Finally, the identified results are displayed virtually overlaid on the near-eye display interface of the augmented reality device for the user to view.

[0023] However, during the actual research and implementation process, the applicant noted that since augmented reality devices are typically head-mounted, there is often a certain deviation between the camera's imaging optical axis and the wearer's visual center. When using the camera to capture images of target objects, it is difficult to accurately determine the alignment between the camera's framing center and the target object. This often requires repeatedly turning the head or adjusting the device's posture to complete the framing, resulting in low operational efficiency. Furthermore, as augmented reality devices increasingly adopt higher-resolution cameras, related solutions typically require processing the entire image. In complex backgrounds or environments with significant lighting changes, this not only results in high computational load and power consumption but also poor stability in target object localization and text recognition.

[0024] In addition, to further improve the recognition accuracy of target objects, complex visual algorithms or additional sensors may be introduced as aids in related technologies. However, this approach increases the complexity and power consumption of augmented reality devices, which is not conducive to the lightweight design and long-term wear of augmented reality devices.

[0025] To overcome the aforementioned shortcomings, the applicant discovered that introducing a laser projector into augmented reality devices, utilizing the linear propagation characteristics of the laser beam emitted by the projector in physical space, allows for the physical indication of the target object by the light spot formed on the surface of the target object. Furthermore, by combining the mapping relationship between the laser projector's emission axis and the camera's imaging optical axis, the position of the light spot in the target object image can be quickly determined when an image acquisition command is detected. This enables recognition processing to be performed only on the local image area corresponding to the light spot, effectively reducing the computational load and improving the stability of target object localization and recognition.

[0026] Based on the above-described inventive concept, the image processing method provided in this application can be generally applied to various augmented reality application scenarios such as document scanning, long-distance information recognition, and assisted travel prompts. It solves problems such as difficulty in framing and alignment, high power consumption in full-frame image processing, and insufficient recognition stability in complex environments in augmented reality devices, significantly improving the information acquisition efficiency and interactive performance of augmented reality devices in practical applications. For ease of understanding, the specific implementation of the image processing method provided in this application will be described below with reference to specific embodiments.

[0027] Figure 1 This is a schematic diagram illustrating the image processing method provided in an embodiment of this application. Figure 1 As shown, this image processing method is applied to an augmented reality device, which includes a laser projector and a camera. The image processing method includes: S110. In response to an image acquisition command, control the camera to acquire an image of a target object containing light spots.

[0028] "Augmented reality device" refers to a head-mounted device that can blend virtual information with real-world scenes for display. Augmented reality devices include at least a laser projector for emitting laser beams and a camera for capturing images of the real-world environment.

[0029] A "laser projector" is an optical device integrated into an augmented reality device. It is used to emit a laser beam outward along the emission axis, so that the laser beam illuminates the surface of the target object and forms a light spot that can be captured by the camera. The laser projector can be a low-power laser that meets safety level requirements, such as a Class 1 safety laser.

[0030] A "camera" refers to an image acquisition module integrated into an augmented reality device, used to image the real environment and generate image data. The camera has a preset imaging optical axis and imaging plane, used to acquire images of target objects, including the light spots formed by the laser beam on the surface of the target object.

[0031] Among them, "image acquisition command" refers to the control command triggered by the user through voice, gesture or head posture, which is used to instruct the camera to perform image acquisition operation.

[0032] "Target object" refers to a physical object in the real world that is located within the field of view of the augmented reality device, is of interest to the user, and is indicated by a laser beam emitted by a laser projector. The target object can reflect or scatter the laser beam, so that a light spot that can be captured by the camera is formed on its surface.

[0033] In some embodiments, the target object can be an object carrying textual or graphic information, including but not limited to paper documents, book pages, signs, electronic displays, traffic signs, billboards, or other physical objects that can be imaged by a camera. Furthermore, the target object can also be other physical targets, such as human bodies, vehicles, buildings, etc.

[0034] "Target object image" refers to an image captured by a camera after responding to an image acquisition command, which includes the light spot formed by the laser beam on the surface of the target object.

[0035] A "spot" refers to a bright spot formed when a laser beam emitted by a laser projector is projected onto the surface of a target object, and its brightness is significantly higher than that of the surrounding area. It is used to indicate the location of the target that the user is interested in.

[0036] In some embodiments, the user triggers an image acquisition command by maintaining a stable head posture for a period of time through voice commands, gesture input, or by detecting the image acquisition command. After detecting the image acquisition command, the augmented reality device controls the camera to perform an image acquisition operation, thereby obtaining an image containing the light spot formed by the laser beam illuminating the surface of the target object.

[0037] S120. Based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera, determine the pixel coordinates of the light spot in the target object image.

[0038] Among them, "pixel coordinates" refers to the two-dimensional position coordinates of the light spot in the image of the target object, which is used to determine the position of the light spot on the camera's imaging plane.

[0039] "Mapping relationship" refers to the correspondence established based on the relative pose between the laser projector and the camera, used to convert the direction of the laser projector's emission axis in physical space into the pixel position in the camera's imaging plane. The relative position offset and relative orientation (such as axis angle, optical center offset, etc.) between the laser projector and the camera can be used as calibration parameters to establish the mapping relationship. The mapping relationship can be obtained through factory calibration and pre-stored in the augmented reality device.

[0040] In some embodiments, during the factory or initialization phase of the augmented reality device, the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera is calibrated through a calibration process, and the mapping relationship is stored in the augmented reality device so that the augmented reality device can predict the corresponding position of the laser beam on the camera imaging plane according to the direction corresponding to the emission axis of the laser projector.

[0041] S130. Based on the pixel coordinates, obtain a local image from the target object image and perform recognition processing on the local image.

[0042] Among them, "local image" refers to an image region of a preset size that is cropped from the image of the target object with pixel coordinates as the center. The local image is a sub-region of the target object image.

[0043] "Recognition processing" refers to the process of analyzing the content information in a local image based on image processing algorithms in order to obtain the recognition result corresponding to the local image.

[0044] In some embodiments, a preset-sized image region is cropped from the target object image as a local image, centered on the pixel coordinates of the light spot in the target object image. Recognition processing is performed only on this local image, without the need for global recognition processing on the entire target object image, thereby reducing the number of pixels involved in the calculation and the number of operations, and improving processing efficiency.

[0045] S140, Display the recognition results.

[0046] The "recognition result" refers to the content information obtained by performing recognition on a local image. The recognition result may include at least one of the following: text information, image content information, and target object information, such as text content, graphic symbols, object category, and object attributes.

[0047] In some embodiments, the identified content is displayed in a magnified form on the near-eye display interface of the augmented reality device, enabling the user to obtain clear identification information while viewing the target object.

[0048] The image processing method of this application embodiment is applied to an augmented reality device. During image acquisition and recognition, by utilizing the mapping relationship between the output axis of the laser projector and the imaging optical axis of the camera, and using the light spot formed by the laser beam on the surface of the target object as a physical indicator, the position of the light spot can be quickly determined in the target object image acquired by the camera when an image acquisition command is detected. This reduces the difficulty of framing and alignment caused by the deviation between the camera's imaging optical axis and the visual center, reduces the interaction cost caused by repeated posture adjustments, and improves the convenience and efficiency of information acquisition and interaction. In addition, this method only performs recognition processing on the local image corresponding to the light spot, without needing to continuously perform high-computation processing such as global feature extraction and sliding window detection on the entire image. This reduces the number of pixels involved in the calculation and the number of operations, thereby reducing processor load and power consumption. At the same time, it can maintain the stability and reliability of the recognition results under complex lighting changes or background interference conditions, which is beneficial to the long-term stable operation of the augmented reality device.

[0049] In some embodiments, such as Figure 2 As shown, S110 specifically includes: S111, Detect the posture change of the augmented reality device.

[0050] "Attitude change" refers to the angular or directional change of an augmented reality device relative to its initial posture. Attitude changes can be detected by gyroscopes, accelerometers, or other attitude sensors.

[0051] In some embodiments, the augmented reality device acquires its own posture data in real time through a built-in posture sensor, and determines whether the augmented reality device has changed its angle relative to its initial posture based on the posture data, in order to detect changes in the posture of the augmented reality device.

[0052] S112. When the posture change meets the preset pointing condition, determine that the laser beam emitted by the laser projector is pointing towards the target object.

[0053] Among them, the "preset pointing condition" refers to the judgment condition used to determine whether the emission direction of the laser projector is basically consistent with the direction of the target object. The preset pointing condition can be set based on the angle range of attitude change or the direction deviation threshold.

[0054] In some embodiments, when the posture change of the augmented reality device is detected to be within a preset angle range and the posture remains stable for a preset time threshold, it is determined that the posture change meets a preset pointing condition, thereby determining that the laser beam emitted by the laser projector is pointing towards the target object.

[0055] S113. When a trigger signal is detected, control the camera to acquire an image of the target object containing the light spot.

[0056] Among them, "trigger signal" refers to the control signal used to instruct the camera to start performing image acquisition operation. The trigger signal can be a command input by the user or a signal automatically generated by the augmented reality device, including but not limited to voice commands, gesture commands, or posture stabilization signals.

[0057] In some embodiments, after the laser beam has been pointed at the target object, when a voice command issued by the user is detected, the camera is controlled to perform an image acquisition operation, thereby acquiring an image of the target object containing the light spot formed by the laser beam on the surface of the target object.

[0058] In this embodiment, firstly, the posture changes of the augmented reality device are continuously detected; secondly, when the posture change is detected to meet the preset pointing conditions, the laser beam emitted by the laser projector is determined to point towards the target object; finally, after the trigger signal is detected, the camera is controlled to acquire an image of the target object containing the light spot. The image acquisition operation can be performed when the pointing conditions and the trigger conditions are met, thereby avoiding false triggering of acquisition and improving the accuracy of image acquisition and interaction efficiency.

[0059] In some embodiments, such as Figure 3 As shown, S120 specifically includes: S121. Obtain the internal parameters of the laser projector.

[0060] "Internal parameters" refer to parameters used to characterize the optical properties of the laser projector itself, including at least focal length and offset angle parameters.

[0061] "Focal length parameter" refers to a parameter used to characterize the relationship between the emission axis of a laser projector and its optical structure, and is used to reflect the directional characteristics of the laser beam during its propagation in space.

[0062] "Offset angle parameter" refers to a parameter used to characterize the angular deviation of the laser projector's output axis relative to the reference coordinate system of the augmented reality device or the imaging optical axis of the camera.

[0063] In some embodiments, during the initialization or factory calibration phase of the augmented reality device, the internal parameters of the laser projector are stored in the augmented reality device for subsequent image processing.

[0064] S122. Based on the internal parameters and the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera, determine the predicted projection position of the emission axis of the laser projector on the imaging plane of the camera.

[0065] Among them, "predicted projection position" refers to the position of the laser projector's output axis on the camera's imaging plane, calculated based on internal parameters and the mapping relationship between the laser projector's output axis and the camera's imaging optical axis.

[0066] In some embodiments, firstly, the first pointing information of the emission axis in the laser projector coordinate system is obtained according to the internal parameters of the laser projector; then, the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera is invoked to transform the first pointing information from the laser projector coordinate system to the camera coordinate system established with the camera imaging optical axis as the reference, thereby obtaining the second pointing information of the emission axis relative to the imaging optical axis in the camera coordinate system; next, with the camera imaging plane as the projection reference and the camera optical center as the projection center, the intersection position of the emission axis and the camera imaging plane is determined according to the second pointing information; then, the pixel position is converted according to the focal length parameter and principal point parameter in the camera intrinsic parameters, thereby obtaining the pixel position corresponding to the intersection position; wherein, the pixel position is the predicted projection position of the emission axis of the laser projector on the camera imaging plane, so as to determine the area where the light spot may appear in the target object image acquired by the camera.

[0067] S123. Based on the predicted projection position and the target object image, determine the candidate pixel positions of the light spot.

[0068] Among them, "candidate pixel position" refers to one or more pixel positions in the target object image that are selected based on the predicted projection position and combined with image features to represent the possible position of the light spot.

[0069] In some embodiments, with the predicted projection position as the center, a corresponding image region is selected in the target object image, and the image features such as brightness, color, or contrast of each pixel in the image region are analyzed to filter out one or more candidate pixel positions that may correspond to the light spot.

[0070] S124. Based on the candidate pixel positions, determine the pixel coordinates of the light spot in the target object image.

[0071] In some embodiments, when there are multiple candidate pixel positions in the target object image, the candidate pixel position with the smallest distance is selected from the multiple candidate pixel positions as the target pixel position based on the distance between each candidate pixel position and the predicted projection position, and the pixel coordinates corresponding to the target pixel position are used as the pixel coordinates of the light spot in the target object image.

[0072] In this embodiment, firstly, the internal parameters of the laser projector are obtained; secondly, based on the internal parameters and the mapping relationship between the laser projector's output axis and the camera's imaging optical axis, the predicted projection position of the laser projector's output axis on the camera's imaging plane is determined; then, combining the predicted projection position and the target object image, one or more candidate pixel positions that may correspond to the light spot are selected in the target object image; finally, the pixel coordinates of the light spot in the target object image are determined from the candidate pixel positions, thereby improving the accuracy of light spot localization and reducing the computational overhead caused by traversing and detecting the candidate pixel positions of the light spot across the entire image.

[0073] In some embodiments, such as Figure 4 As shown, S123 specifically includes: S1231. Using the predicted projection position as the center point, extract a first local image of a first preset size from the target object image.

[0074] Here, "first local image" refers to an image region of a first preset size that is cropped from the image of the target object with the predicted projection position as the center.

[0075] The "first preset size" is used to limit the search range of candidate spot positions. Its size can be set according to factors such as the divergence angle and calibration error of the laser projector, so as to reduce the interference of irrelevant background areas on spot detection while ensuring that the first local image contains the spot.

[0076] In some embodiments, after determining the predicted projection position of the laser projector's output axis on the camera's imaging plane, an image region of a first preset size is selected in the target object image centered on the predicted projection position as a candidate search region for subsequent spot detection.

[0077] S1232. Perform brightness analysis on each pixel in the first local image and filter out pixels whose brightness is higher than a preset brightness threshold.

[0078] Among them, "brightness analysis" refers to analyzing and processing the brightness features of each pixel in the first local image to obtain the numerical information of the pixel in the brightness dimension.

[0079] "Preset brightness threshold" refers to the brightness threshold used to distinguish laser spot pixels from background pixels. This preset brightness threshold can be set according to the imaging environment or the parameters of the augmented reality device.

[0080] In some embodiments, by comparing the brightness value of a pixel in a first local image with a preset brightness threshold, pixels with a brightness higher than the preset brightness threshold can be filtered out, thereby excluding most background pixels that are not laser spots.

[0081] S1233. Based on the location information of the selected pixels, determine at least one brightness concentration area.

[0082] The “brightness concentration area” refers to a region in the first local image that is composed of multiple pixels whose brightness is higher than a preset brightness threshold and which are concentrated in spatial location.

[0083] In some embodiments, spatial clustering or connected component analysis is performed on the selected bright pixels to group spatially adjacent or connected bright pixels into the same pixel set. For example, if multiple bright pixels form continuously distributed bright spots in an image, they are grouped into a single brightness concentration region.

[0084] S1234. Calculate the center position of each concentrated brightness region.

[0085] Here, "center position" refers to the geometric center calculated based on the position information of each pixel within the brightness concentration area.

[0086] In some embodiments, the average of the horizontal and vertical coordinates of all pixels in each brightness concentration region is calculated to obtain the geometric center position of the brightness concentration region; or a weighted average method is used to calculate the brightness centroid position with pixel brightness as the weight, thereby more accurately reflecting the true center of the light spot.

[0087] S1235. The center position is determined as the candidate pixel position of the light spot.

[0088] In this embodiment, firstly, a first local image of a first preset size is cropped with the predicted projection position as the center, thereby limiting the possible distribution area of ​​the light spot within the first local image; secondly, brightness analysis is performed on each pixel in the first local image, and pixels with brightness higher than a preset brightness threshold are selected, and clustering analysis is performed based on the spatial position information of the selected pixels to determine at least one brightness concentration region; finally, the center position of each brightness concentration region is calculated, and the center position is determined as the candidate pixel position of the light spot, thereby achieving accurate positioning of the light spot within the first local image.

[0089] In some embodiments, such as Figure 5 As shown, S124 specifically includes: S1241. When there are multiple candidate pixel positions in the target object image, determine the distance between each candidate pixel position and the predicted projection position.

[0090] Here, "distance" refers to a metric used to characterize the spatial proximity between the candidate pixel position and the predicted projection position.

[0091] In some embodiments, when there are multiple candidate pixel locations in the target object image, the distance between each candidate pixel location and the predicted projection location of the laser projector's output axis on the camera's imaging plane is calculated to evaluate the spatial proximity between each candidate pixel location and the predicted location.

[0092] S1242. From multiple candidate pixel positions, select the candidate pixel position with the smallest distance from the predicted projection position as the target pixel position.

[0093] The "target pixel position" refers to the pixel position that is closest to the predicted projection position, selected from multiple candidate pixel positions based on the distance between each candidate pixel position and the predicted projection position.

[0094] In some embodiments, after comparing the distance between each candidate pixel position and the predicted projection position, the candidate pixel position with the smallest distance is selected from multiple candidate pixel positions and determined as the target pixel position, so as to improve the consistency between the spot localization result and the prediction result.

[0095] S1243. The pixel coordinates corresponding to the target pixel position are used as the pixel coordinates of the light spot in the target object image.

[0096] In some embodiments, after comparing the distance between each candidate pixel position and the predicted projection position, the candidate pixel position corresponding to the smallest distance pixel is selected from multiple candidate pixel positions as the target pixel coordinates, and the target pixel coordinates are determined as the final pixel coordinates of the light spot in the target object image, which are then used to perform local image cropping, text recognition, and display processing operations based on the pixel coordinates.

[0097] In this embodiment, firstly, when there are multiple candidate pixel positions in the target object image, the distance between each candidate pixel position and the predicted projection position is determined; secondly, the distances between each candidate pixel position and the predicted projection position are compared, and the candidate pixel position with the smallest distance from the predicted projection position is selected as the target pixel position; finally, the pixel coordinates corresponding to the target pixel position are used as the pixel coordinates of the light spot in the target object image. This method can accurately select the pixel position that best matches the predicted projection position from multiple candidate pixel positions, thereby improving the reliability of light spot localization.

[0098] In some embodiments, such as Figure 6 As shown, S130 specifically includes: S131. Using the pixel coordinates of the light spot in the target object image as the center point, extract a second partial image of a second preset size from the target object image.

[0099] The "second preset size" refers to the size parameter used to determine the cropping range of the second local image. The second preset size can be set according to the size of the target object image, and the second preset size is less than or equal to the first preset size.

[0100] The "second local image" refers to the image region extracted from the target object image according to a preset size, with the pixel coordinates of the light spot in the target object image as the center point. The local image is a sub-region of the target object image.

[0101] In some embodiments, after determining the pixel coordinates (u, v) of the light spot in the target object image, an image region of a second preset size is cropped from the target object image with (u, v) as the center point, as a local image. For example, a sub-image region of 512×512 pixels or 800×800 pixels is cropped so that only this local region is processed subsequently.

[0102] S132. Perform resolution enhancement processing on the second local image to obtain an enhanced image.

[0103] "Enhanced image" refers to the image obtained after performing resolution enhancement processing on a local image. The enhanced image has a higher effective resolution or clearer details compared to the local image.

[0104] In some embodiments, the captured local image is subjected to resolution enhancement processing. Through super-resolution reconstruction, interpolation enhancement, or neural network-based image enhancement algorithms, the clarity of text edges and details in the local image is improved, thereby generating a higher resolution enhanced image.

[0105] S133. Perform recognition processing on the enhanced image to obtain recognition information in the enhanced image.

[0106] "Identification information" refers to the content data obtained by recognizing the enhanced image, including at least one of the following: text information, image content information, and target object information.

[0107] In some embodiments, optical character recognition (OCR) processing is performed on the enhanced image after resolution enhancement to extract the characters, words or sentence content contained therein, and the recognized text information is output, such as recognizing the prompt text "Construction ahead, please detour" on the display screen for subsequent display.

[0108] In this embodiment, firstly, a second local image of a second preset size is extracted from the target object image, centered on the pixel coordinates of the light spot in the target object image; secondly, the second local image is subjected to resolution enhancement processing to generate an enhanced image with higher resolution; finally, the enhanced image is subjected to recognition processing to extract the recognition information contained in the enhanced image. The enhancement and recognition processing is performed only on the local image area indicated by the laser beam, which can improve the recognition accuracy in long-distance or small-sized text scenes while reducing the overall computing load and power consumption.

[0109] In some embodiments, such as Figure 7 As shown, S140 specifically includes: S141. Based on the principle of rectilinear propagation of light, determine the projection display position of the laser beam emitted by the laser projector on the preset projection plane.

[0110] The "principle of rectilinear propagation of light" refers to the physical property of light propagating in a straight line in a uniform medium, and is used to characterize the propagation direction of a laser beam in space after it is emitted from a laser projector.

[0111] "Preset projection plane" refers to a reference plane predefined in augmented reality devices to describe the projection position of the laser beam. The preset projection plane is used to determine the projection position of the laser beam in physical space.

[0112] "Projection display position" refers to the position coordinates of the laser beam emitted by the laser projector on a preset projection plane, determined based on the principle of rectilinear propagation of light.

[0113] In some embodiments, based on the physical property that light propagates in a straight line in a uniform medium, the emission starting point and emission direction of the laser beam are obtained, and the laser beam is regarded as a straight line extending along the emission direction; further, the intersection point of the straight line and the preset projection plane is calculated, and the intersection point is determined as the projection display position of the laser beam emitted by the laser projector on the preset projection plane.

[0114] S142. Map the projected display position to the near-eye display coordinate system of the augmented reality device according to a preset mapping relationship to obtain the corresponding target display position.

[0115] The "near-eye display coordinate system" refers to the coordinate system used in augmented reality devices to describe the position of the content displayed by the near-eye display module, and is used to determine the display position of virtual content in the user's field of vision.

[0116] "Target display position" refers to the position obtained by mapping the projected display position to the near-eye display coordinate system according to a preset mapping relationship. It is used to indicate the presentation position of virtual content in the display interface of augmented reality devices.

[0117] In some embodiments, based on a pre-established mapping relationship between the laser projector and the near-eye display module, the projection display position of the laser beam on a preset projection plane is mapped to the near-eye display coordinate system, thereby determining the target display position corresponding to the projection display position in the display interface of the augmented reality device.

[0118] S143. Display the magnified enhanced image and the identification information at the target display position.

[0119] In some embodiments, after determining the display position, the magnified enhanced image is displayed in the form of a floating window or a label box, and the corresponding text recognition result is displayed in or adjacent to the enhanced image, so that the user can intuitively obtain clear image content and text information while observing the target object.

[0120] In this embodiment, firstly, based on the principle of rectilinear propagation of light, the projection display position of the laser beam emitted by the laser projector on the preset projection plane is determined; secondly, according to the mapping relationship between the laser projector and the near-eye display module of the augmented reality device, the projection display position is mapped to the near-eye display coordinate system to obtain the corresponding target display position; finally, the magnified augmented image and corresponding text information are displayed at the target display position, so that the text recognition result is consistent with the spatial position of the target object in the user's field of vision, thereby improving the intuitiveness of information presentation.

[0121] Figure 8 This is a schematic diagram of the architecture of an image processing apparatus provided in an embodiment of this application. Figure 8 As shown, the image processing device 200 includes: a control module 210, a determination module 220, a recognition module 230, and a display module 240, wherein: Control module 210 is used to control the camera to acquire an image of a target object containing a light spot in response to an image acquisition command, wherein the light spot is a bright spot formed by the laser beam emitted by the laser projector projected onto the surface of the target object; The determining module 220 is used to determine the pixel coordinates of the light spot in the target object image based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera; The recognition module 230 is used to obtain a local image from the target object image based on the pixel coordinates, and to perform recognition processing on the local image; Display module 240 is used to display the recognition results.

[0122] In some embodiments, the control module 210 is specifically used to: detect the posture change of the augmented reality device; when the posture change meets a preset pointing condition, determine that the laser beam emitted by the laser projector is pointing towards the target object; when a trigger signal is detected, control the camera to acquire an image of the target object containing a light spot; wherein, the preset pointing condition is used to indicate a determination condition that the emission direction of the laser projector is basically consistent with the direction of the target object.

[0123] In some embodiments, the determining module 220 includes: an acquisition unit, a first determining unit, a second determining unit, and a third determining unit, wherein: the acquisition unit is used to acquire the internal parameters of the laser projector; the first determining unit is used to determine the predicted projection position of the laser projector's output axis on the camera's imaging plane based on the internal parameters and the mapping relationship between the laser projector's output axis and the camera's imaging optical axis; the second determining unit is used to determine the candidate pixel position of the light spot based on the predicted projection position and the target object image; the third determining unit is used to determine the pixel coordinates of the light spot in the target object image based on the candidate pixel position; wherein the internal parameters include at least a focal length parameter and an offset angle parameter.

[0124] In some embodiments, the second determining unit is specifically configured to: extract a first local image of a first preset size from the target object image with the predicted projection position as the center point; perform brightness analysis on each pixel in the first local image and filter out pixels with brightness higher than a preset brightness threshold; determine at least one brightness concentration region based on the position information of the filtered pixels; calculate the center position of each brightness concentration region; and determine the center position as the candidate pixel position of the light spot.

[0125] In some embodiments, the third determining unit is specifically used to: when there are multiple candidate pixel positions in the target object image, determine the distance between each candidate pixel position and the predicted projection position; select the candidate pixel position with the smallest distance to the predicted projection position from the multiple candidate pixel positions as the target pixel position; and use the pixel coordinates corresponding to the target pixel position as the pixel coordinates of the light spot in the target object image.

[0126] In some embodiments, the recognition module 230 is specifically configured to: extract a second partial image of a second preset size from the target object image, using the pixel coordinates of the light spot in the target object image as the center point; perform resolution enhancement processing on the second partial image to obtain an enhanced image; and perform recognition processing on the enhanced image to obtain recognition information in the enhanced image; wherein the recognition information includes at least one of text information, image content information, and target object information.

[0127] In some embodiments, the display module 240 is specifically used to: determine the projection display position of the laser beam emitted by the laser projector on a preset projection plane based on the principle of rectilinear propagation of light; map the projection display position to the near-eye display coordinate system of the augmented reality device according to a preset mapping relationship to obtain the corresponding target display position; and display the magnified augmented image and the recognition information at the target display position.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the functional modules described above can be referred to the corresponding method steps in the foregoing embodiments, and will not be repeated here. Those skilled in the art can use different methods to implement the described functions for each specific application. For example, a computer software program containing the steps of the above method embodiments can be stored in a computer-readable storage medium so that, when executed, the program can implement one or more steps of the above method embodiments.

[0129] Figure 9 The diagram shows a schematic representation of an augmented reality device according to an embodiment of this application. This embodiment does not limit the specific implementation of the augmented reality device.

[0130] like Figure 9 As shown, the augmented reality device 300 may include: a laser projector 310, a camera 320, a processor 330, a communication interface 340, a memory 350, and a communication bus 360.

[0131] The system includes a laser projector 310 for emitting a laser beam toward a target object to form a light spot on the object's surface; a camera 320 for acquiring an image of the target object containing the light spot; a processor 330 communicatively connected to the laser projector 310, the camera 320, and the processor 330, the communication interface 340, and the memory 350 communicating with each other via a communication bus 360. The communication interface 340 is used for communication with other external devices. The processor 330 executes a program 370 to implement the image processing method described in one or more of the above embodiments.

[0132] Specifically, program 370 may include program code that includes computer operation instructions. When program 370 is invoked, processor 330 executes the computer operation instructions to implement the steps in the image processing method of one or more embodiments.

[0133] Depending on the actual application scenario, the processor 330 can be of the appropriate type, including but not limited to mainstream embedded processors such as microcontrollers (MCUs) and ARM architecture processors, as well as other types of processors such as digital signal processors (DSPs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs), as long as they can provide the computing and control capabilities required by the actual application scenario.

[0134] Memory 350 is used to store program 370. It includes a program storage area and a data storage area. The program storage area is used to store firmware programs, embedded applications, and various functional modules; the data storage area is used to store data and calculation results during program execution. For example, memory 350 may include: on-chip RAM (for temporary data storage during program execution); on-chip Flash memory (for storing program code and configuration data); and EEPROM or other types of non-volatile memory (for storing parameters that need to be retained when power is off).

[0135] This application also provides a computer-readable storage medium. This computer-readable storage medium can be a non-volatile computer-readable storage medium. This computer-readable storage medium stores a computer program.

[0136] When executed by a processor, the computer program implements one or more steps of the image processing method disclosed in the embodiments of this application. A complete computer program product is embodied on one or more computer-readable storage media containing the computer program disclosed in the embodiments of this application.

[0137] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above. For the sake of brevity, they are not provided in detail. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method applied to an augmented reality device, characterized in that, The augmented reality device includes a laser projector and a camera, and the method includes: In response to an image acquisition command, the camera is controlled to acquire an image of a target object containing a light spot, wherein the light spot is a bright spot formed by the laser beam emitted by the laser projector projected onto the surface of the target object; Based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera, the pixel coordinates of the light spot in the target object image are determined; Based on the pixel coordinates, a local image is obtained from the target object image, and the local image is subjected to recognition processing. Display the recognition results.

2. The image processing method of claim 1, wherein, The step of controlling the camera to acquire an image of a target object containing light spots in response to an image acquisition command includes: Detect the pose changes of the augmented reality device; When the posture change meets the preset pointing condition, it is determined that the laser beam emitted by the laser projector is pointing towards the target object; When a trigger signal is detected, the camera is controlled to acquire an image of the target object containing the light spot; The preset pointing condition is used to indicate the determination condition that the emission direction of the laser projector is basically consistent with the direction of the target object.

3. The image processing method of claim 1, wherein, Determining the pixel coordinates of the light spot in the target object image based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera includes: Obtain the internal parameters of the laser projector; Based on the internal parameters and the mapping relationship between the output axis of the laser projector and the imaging optical axis of the camera, the predicted projection position of the output axis of the laser projector on the imaging plane of the camera is determined. Based on the predicted projection position and the target object image, the candidate pixel positions of the light spot are determined; Based on the candidate pixel positions, the pixel coordinates of the light spot in the target object image are determined; The internal parameters include at least the focal length parameter and the offset angle parameter.

4. The image processing method according to claim 3, characterized in that, Determining the candidate pixel positions of the light spot based on the predicted projection position and the target object image includes: Using the predicted projection position as the center point, a first local image of a first preset size is extracted from the target object image; Brightness analysis is performed on each pixel in the first local image, and pixels with brightness higher than a preset brightness threshold are selected. Based on the location information of the selected pixels, at least one brightness concentration area is determined; Calculate the center position of each concentrated brightness region; The center position is determined as the candidate pixel position of the light spot.

5. The image processing method according to claim 3, characterized in that, Determining the pixel coordinates of the light spot in the target object image based on the candidate pixel positions includes: When there are multiple candidate pixel locations in the target object image, the distance between each candidate pixel location and the predicted projection location is determined respectively; From multiple candidate pixel positions, the candidate pixel position with the smallest distance from the predicted projection position is selected as the target pixel position; The pixel coordinates corresponding to the target pixel position are used as the pixel coordinates of the light spot in the target object image.

6. The image processing method according to claim 1, characterized in that, The step of obtaining a local image from the target object image based on the pixel coordinates and performing recognition processing on the local image includes: Using the pixel coordinates of the light spot in the target object image as the center point, a second partial image of a second preset size is cropped from the target object image; The second local image is subjected to resolution enhancement processing to obtain an enhanced image; The enhanced image is subjected to recognition processing to obtain recognition information in the enhanced image; The identification information includes at least one of text information, image content information, and target object information.

7. The image processing method according to claim 6, characterized in that, The displayed recognition results include: Based on the principle of rectilinear propagation of light, the projection display position of the laser beam emitted by the laser projector on the preset projection plane is determined; The projected display position is mapped to the near-eye display coordinate system of the augmented reality device according to a preset mapping relationship to obtain the corresponding target display position; An enhanced, magnified image and the identification information are displayed at the target display location.

8. An image processing apparatus, characterized in that, include: The control module is used to control the camera to acquire an image of a target object containing a light spot in response to an image acquisition command, wherein the light spot is a bright spot formed by a laser beam emitted by a laser projector projected onto the surface of the target object; The determination module is used to determine the pixel coordinates of the light spot in the target object image based on the mapping relationship between the emission axis of the laser projector and the imaging optical axis of the camera; The recognition module is used to obtain a local image from the target object image based on the pixel coordinates, and to perform recognition processing on the local image; The display module is used to display the recognition results.

9. An augmented reality device, characterized in that, include: A laser projector is used to emit a laser beam toward a target object to form a light spot on the surface of the target object; A camera is used to acquire an image of the target object containing the light spot; A processor, which is communicatively connected to the laser projector and the camera; A memory, communicatively connected to the processor, stores computer program instructions that, when invoked by the processor, cause the processor to execute the image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores processor-executable computer program instructions, which, when executed by the processor, cause the computer to perform the image processing method as described in any one of claims 1-7.

Images