Visual following method, apparatus, device, storage medium and program product

Abstract

This application relates to the field of computer technology, providing a visual tracking method, apparatus, device, storage medium, and program product. The method includes: performing exposure compensation processing on the image to be processed based on the real illumination information at the time of shooting; performing smoothing filtering processing on the exposure-compensated image to be processed based on the illumination change rate at the time of shooting; wherein the illumination change rate is predicted before shooting the image to be processed; and performing target tracking based on the smoothed image to be processed. This application can effectively improve the accuracy of target recognition during target tracking, thereby improving the success rate of visual tracking in uneven lighting scenes.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a visual following method, apparatus, device, storage medium, and program product. Background Technology

[0002] Visual tracking is one of the core technologies for embodied robots to achieve advanced functions such as human-computer interaction and autonomous movement. This technology enables robots to continuously lock onto and track specific targets (such as family members or guides) through visual sensors, and is key to applications in smart home services, elderly care, and child companionship. The visual tracking technology of embodied robots relies on target detection and tracking algorithms in computer vision, and its core processes include image acquisition, feature extraction, target matching, and motion control.

[0003] Although existing visual tracking technology has achieved good results under ideal lighting conditions, in scenarios such as smart homes, the presence of various complex and variable lighting conditions makes it impossible to guarantee the tracking success rate in uneven lighting scenarios. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the related art. To this end, this application proposes a visual following method, apparatus, device, storage medium, and program product to solve the problem that the success rate of visual following cannot be guaranteed in uneven lighting scenes, and to improve the success rate of visual following in uneven lighting scenes.

[0005] The visual following method according to the first aspect of this application includes: Based on the real illumination information at the time the image to be processed was captured, exposure compensation processing is performed on the image to be processed. Based on the illumination change rate at the time of image capture, the image to be processed after exposure compensation is subjected to smoothing filtering; the illumination change rate is predicted before the image to be processed is captured. Target tracking is performed based on the image to be processed after smoothing filtering.

[0006] According to one embodiment of this application, the target tracking based on the image to be processed after smoothing filtering includes: Visual features are extracted from the image to be processed after smoothing filtering to obtain visual features; Geometric features are extracted from the image to be processed after smoothing filtering to obtain geometric features; The feature fusion weights are determined based on the rate of change in illumination. Based on the feature fusion weights, the visual features and the geometric features are fused to obtain fused features; Target following is performed based on the fused features.

[0007] According to one embodiment of this application, the exposure compensation processing of the image to be processed based on the real illumination information at the time of shooting the image includes: Determine the brightness range of the actual illumination information at the time the image to be processed was captured; The exposure compensation method is determined based on the brightness range; wherein, each brightness range corresponds to one exposure compensation method; the exposure compensation method is one of image compression, no processing, and image enhancement; The exposure compensation process is applied to the image to be processed based on the aforementioned exposure compensation method.

[0008] According to one embodiment of this application, the exposure compensation processing of the image to be processed based on the exposure compensation method includes any one of the following: If the exposure compensation method is image compression, then gamma compression is performed on the image to be processed; If the exposure compensation method is no processing, then no compensation will be performed on the image to be processed; If the exposure compensation method is image enhancement, then the image to be processed is optimized for both illumination and reflection components.

[0009] According to one embodiment of this application, the reflection component optimization of the image to be processed includes: Edge detection is performed on the image to be processed to obtain the edge detection result; Based on the edge detection results, the reflection component of the image to be processed is optimized.

[0010] According to one embodiment of this application, the smoothing filtering process performed on the exposure-compensated image based on the illumination change rate at the time of image capture includes: The smoothing filter factor is determined based on the rate of change of illumination at the time of shooting the image to be processed. Based on the smoothing filter factor, the image to be processed after exposure compensation is subjected to smoothing filter processing.

[0011] A visual following device according to a second aspect embodiment of this application includes: The exposure compensation module is used to perform exposure compensation processing on the image to be processed based on the real illumination information at the time of shooting. A smoothing filter module is used to perform smoothing filter processing on the image to be processed after exposure compensation based on the illumination change rate at the time of shooting; the illumination change rate is predicted before shooting the image to be processed. The target following module is used to perform target following based on the image to be processed after smoothing filtering.

[0012] An electronic device according to a third aspect of this application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual following method as described above.

[0013] According to a fourth aspect of this application, the storage medium is a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the visual following method as described above.

[0014] A computer program product according to a fifth aspect of this application includes a computer program that, when executed by a processor, implements the visual following method as described above.

[0015] The above-described one or more technical solutions in the embodiments of this application have at least the following technical effects: Based on the actual illumination information at the time the image was captured, exposure compensation is applied to prevent overexposure or underexposure, thus preventing the loss of image details. Furthermore, based on the illumination change rate at the time the image was captured, a smoothing filter is applied to the image after exposure compensation. Since the illumination change rate is predicted before the image was captured, it reduces the interference of drastic illumination changes on target tracking. Therefore, target tracking based on the smoothed image can effectively improve the accuracy of target recognition during target tracking, thereby increasing the success rate of visual tracking in uneven lighting scenarios.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the visual following method provided in the embodiments of this application.

[0019] Figure 2 This is the second flowchart of the visual following method provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application proposes a visual following method, apparatus, device, storage medium, and program product. Specifically, it proposes a visual following method based on a three-loop collaborative approach of "illumination-feature-control," achieving stable following in uneven lighting scenarios through dynamic exposure compensation, illumination transient smoothing, and multimodal feature fusion. Because the visual following method of this application has a lightweight structure and low resource consumption, it is suitable for consumer-grade robotic dogs and other embodied robots.

[0023] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0024] Figure 1 This is one of the flowcharts illustrating the visual following method provided in the embodiments of this application, such as... Figure 1 As shown, the visual following method includes: Step 110: Based on the real lighting information at the time the image was captured, perform exposure compensation processing on the image to be processed.

[0025] Step 120: Based on the illumination change rate at the time of image capture, perform smoothing filtering on the image after exposure compensation processing; the illumination change rate is predicted before capturing the image.

[0026] Step 130: Perform target tracking based on the image to be processed after smoothing filtering.

[0027] It should be noted that the execution subject of the visual following method provided in this application embodiment can be a robot, more specifically a embodied robot, such as a humanoid robot, a robot dog, etc.

[0028] Specifically, in addition to the modules that realize the functions required by the robot, the robot of this application may also be equipped with an image acquisition module (such as a camera, webcam, etc.) and a light sensor.

[0029] Among them, the light sensor can collect external light information in real time, such as real-time light information of the area corresponding to the image acquisition module.

[0030] The image acquisition module can acquire images of the area at regular or irregular intervals, or it can store multiple frames of images through video recording. Specifically, the images in this application can be monocular RGB images.

[0031] Based on this, in this application, the robot can obtain the real lighting information at the time of image capture from the lighting information collected by the light sensor each time it captures or acquires an image.

[0032] Furthermore, the robot can follow targets based on each image.

[0033] It should be noted that, in this application, the robot can preprocess the image before following the target based on the image to improve the success rate of following.

[0034] Therefore, this application can treat each unprocessed image as a single image to be processed.

[0035] Figure 2 This is a second flowchart illustrating the visual following method provided in the embodiments of this application, as shown below. Figure 2 As shown, the robot of this application may be equipped with an exposure compensation module (which may also be referred to as a dynamic exposure compensation module), a smoothing filter module (which may also be referred to as a lighting transient smoothing module), and a target following module. The target following module may include a feature fusion module and a following module (not explicitly shown).

[0036] Understandably, in this application, any image to be processed can be input into the dynamic exposure compensation module along with its corresponding real lighting information.

[0037] Furthermore, the dynamic exposure compensation module can determine whether exposure compensation is needed for the image to be processed based on the actual lighting information at the time the image was captured.

[0038] In one embodiment, to avoid misjudgment due to single noise, this application may perform exposure compensation on the current image to be processed only when the real illumination information corresponding to multiple consecutive frames of images (including the current image) meets the specified conditions.

[0039] The exposure compensation in this application can automatically constrict the pupil in strong light (corresponding to the image compression in this application) to avoid saturation of retinal photoreceptor cells, based on the dynamic adjustment mechanism of the human eye pupil; and automatically dilate the pupil in low light (corresponding to the image enhancement in this application) to enhance the perception of details in dark areas.

[0040] Therefore, the dynamic exposure compensation module can determine whether to perform image compression, image enhancement, or no exposure compensation processing on the image to be processed based on the real lighting information at the time the image was captured.

[0041] Based on the above exposure compensation processing, this application can achieve three-area collaborative work, enabling the robot to clearly identify targets in both midday sunlight and indoor shadows.

[0042] After performing exposure compensation processing (including image compression, image enhancement, or no processing) on ​​the image to be processed, the dynamic exposure compensation module can output the resulting image to the illumination transient smoothing module.

[0043] It should be noted that the illumination transient smoothing module of this application can be equipped with a pre-trained Long Short-Term Memory (LSTM) network model as the illumination prediction model. Specifically, this application can use a 3-layer LSTM model (input dimension 3, hidden layer dimension 64, output dimension 2), thereby predicting illumination based on the previous 3 frames of illumination sequence. , , Predicting future light levels With the rate of change of light .

[0044] In this application, the LSTM model employs the Adam optimizer (learning rate of 0.001) during training and uses an early stopping strategy (training stops if the validation set loss does not decrease after 3 epochs). It is trained for 100 epochs on a dataset of 100,000 frames of lighting sequences. The resulting LSTM model, serving as a lighting prediction model, can effectively predict future lighting changes, facilitating subsequent image smoothing. The Adam optimizer is a widely used adaptive learning rate optimization algorithm in deep learning. The 100 epochs represent the model's complete traversal of the entire 100,000-frame dataset 100 times, allowing it to fully learn the patterns within the data.

[0045] Therefore, the illumination transient smoothing module can input the illumination sequence preceding the image to be processed (e.g., the illumination sequence formed by the real illumination information corresponding to the three frames preceding the image to be processed) into the illumination prediction model in advance, and obtain the illumination prediction value and illumination change rate corresponding to the shooting time of the image to be processed output by the illumination prediction model.

[0046] By designing an LSTM model, the rate of change of illumination can be predicted. When the rate of change is high, smoothing filtering is triggered in advance, and the filter coefficient is dynamically adjusted according to the predicted changes in illumination, making the tracking process more stable and smooth.

[0047] It should be noted that when there are fewer than three frames preceding the image to be processed, the actual illumination information from the only one or two frames can be used to form an illumination sequence for prediction, or prediction can be temporarily omitted. In the absence of an illumination change rate, it can be assumed that the illumination change rate is relatively slow, and a larger smoothing filter factor can be used when determining the smoothing filter factor.

[0048] Furthermore, a smoothing filter factor is determined based on the specific rate of change of illumination, and this smoothing filter factor is used to smooth the input image to be processed (i.e., the image to be processed after exposure compensation). Specifically, a smaller smoothing filter factor is used when the rate of change of illumination changes drastically, while a larger smoothing filter factor is used when the rate of change of illumination changes more slowly.

[0049] Therefore, by applying smoothing filtering to the image to be processed, the problem of target tracking failure in scenes with drastic changes in lighting can be effectively alleviated.

[0050] Furthermore, the illumination transient smoothing module can output the smoothed and filtered image to the target following module. The target following module, in cooperation with the feature fusion module and the following model, performs operations such as feature extraction, feature fusion, target detection, and target following, thereby enabling continuous target following.

[0051] According to the visual tracking method of this application embodiment, exposure compensation processing is performed on the image to be processed based on the real illumination information at the time of shooting, which can prevent overexposure or underexposure of the image to be processed, thereby preventing the loss of image details; and, based on the illumination change rate at the time of shooting, smoothing filtering processing is performed on the image to be processed after exposure compensation processing. Since the illumination change rate is predicted before shooting the image to be processed, the interference of drastic illumination changes on target tracking can be reduced. Therefore, target tracking based on the image to be processed after smoothing filtering can effectively improve the accuracy of target recognition during target tracking, thereby improving the success rate of visual tracking in uneven illumination scenes.

[0052] In one embodiment, exposure compensation processing is performed on the image to be processed based on the actual illumination information at the time the image was captured, including: Determine the brightness range of the actual illumination information at the time the image to be processed was captured; The exposure compensation method is determined based on the brightness range; each brightness range corresponds to one exposure compensation method; the exposure compensation method is one of image compression, no processing, and image enhancement. Exposure compensation processing is performed on the image to be processed based on the exposure compensation method.

[0053] Specifically, this application allows for the preset of multiple brightness ranges, such as overly bright (>5000 lux), normal (500-5000 lux), and overly dark (<500 lux).

[0054] Furthermore, an exposure compensation method is set for each brightness range. Specifically, for excessively bright brightness ranges, the exposure compensation method of "image compression" is associated; for normal brightness ranges, the exposure compensation method of "no processing" is associated; and for excessively dark brightness ranges, the exposure compensation method of "image enhancement" is associated.

[0055] Therefore, in this application, the actual illumination information at the time the image to be processed was captured can be compared with the preset brightness ranges to determine which brightness range the actual illumination information at the time the image to be processed was in.

[0056] After determining the brightness range, you can further search for the corresponding exposure compensation method based on the brightness range, and then perform corresponding exposure compensation processing on the image to be processed according to the found exposure compensation method.

[0057] In one embodiment, after determining the brightness range of the current image to be processed, this application can further query whether the actual illumination information at the shooting time of the two consecutive frames preceding the image to be processed is also within the same brightness range as the current image to be processed. If so, the corresponding exposure compensation method is further searched based on the brightness range, and then the image to be processed is subjected to corresponding exposure compensation processing according to the found exposure compensation method. If not, the same exposure compensation method as the previous frame is used to perform exposure compensation processing on the image to be processed.

[0058] If the image to be processed is the first frame or the second frame, then the exposure compensation process for the image to be processed can be performed in the same way as for the first frame.

[0059] In another embodiment, this application can also determine the exposure compensation method and perform exposure compensation processing based on the brightness range corresponding to the threshold when the real illumination information corresponding to three consecutive frames of images all meet the same threshold.

[0060] The thresholds can include an excessively bright threshold and an excessively dark threshold. The excessively bright threshold = 5000 lux + ambient light fluctuation coefficient (±500 lux), and the excessively dark threshold = 500 lux - ambient light fluctuation coefficient (±50 lux).

[0061] This application performs exposure compensation processing on the image to be processed based on the real illumination information at the time of shooting, which can prevent the image to be processed from being overexposed or underexposed, thereby preventing the loss of image details, helping to improve the accuracy of target recognition during target tracking, and thus improving the success rate of visual tracking in uneven lighting scenes.

[0062] In one embodiment, exposure compensation processing of the image to be processed based on an exposure compensation method includes any one of the following: If the exposure compensation method is image compression, then gamma compression is applied to the image to be processed; If the exposure compensation method is set to no processing, then no compensation will be applied to the image to be processed. If the exposure compensation method is image enhancement, then the image to be processed will be optimized for both the illumination component and the reflection component.

[0063] Specifically, if the exposure compensation method is determined to be no processing, then no exposure compensation will be performed on the image to be processed, and the image to be processed can be directly output to the illumination transient smoothing module.

[0064] If the exposure compensation method is determined to be image compression, then gamma compression is applied to the image to be processed.

[0065] It should be noted that when the environment is too bright, it can cause the sensor to saturate and overexpose. Therefore, gamma compression needs to be performed on the image to be processed in overly bright areas to prevent overexposure.

[0066] Specifically, gamma compression of the image to be processed can be performed using the following formula: ; ; in, This represents the compressed image to be processed. This represents the image to be processed before compression. This represents the actual lighting information at the time the image to be processed was captured.

[0067] As shown in the above formula, the image to be processed is first normalized and then compressed with gamma. After compression, it is denormalized to obtain the original image.

[0068] in the formula γ The compression ratio can be adaptively adjusted based on the actual lighting information. When the actual lighting information is less than 5000 lux... γ >1.0 (uncompressed), 0.6 < γ < 1.0 (automatically adjusted compression ratio) when the actual lighting information is between 5000-10000 lux, and 0.6 < γ < 1.0 when the actual lighting information is greater than 10000 lux. γ=0.6 (maximum compression).

[0069] This application addresses the shortcomings of traditional exposure algorithms by designing a dynamic gamma compression curve, which reduces the subjective brightness mapping error to less than 10%, solves the problem of local overexposure caused by global correction, and improves the accuracy of target detection.

[0070] If the exposure compensation method is determined to be image enhancement, then the image to be processed will be optimized for both the illumination component and the reflection component.

[0071] It should be noted that, according to Retinex theory, an image can be decomposed into illumination components. With reflection component ,Right now: ; in, This represents the image to be processed. This represents the illumination components of the image to be processed. This represents the reflection component of the image to be processed.

[0072] Retinex is a portmanteau of "Retina" and "Cortex," used to explain how humans perceive color and brightness.

[0073] Therefore, when the exposure compensation method is image enhancement, this application can optimize the illumination component and the reflection component of the image to be processed separately.

[0074] The illumination component represents the light intensity distribution in the scene, that is, the influence of ambient light on the image, describing the distribution and intensity changes of light in the image. Therefore, when optimizing the illumination component of the image to be processed, this application can specifically use a three-scale Gaussian filter to separate the illumination component: ; in, It is a Gaussian filter kernel (the standard deviation of the Gaussian distribution is σ = 3 / 5 / 7). I This represents the image to be processed, specifically the image before enhancement; 0.5, 0.3, and 0.2 are all weights, which can be optimized using the entropy weight method to ensure that the illumination components are smooth while preserving edge information, effectively solving the halo problem caused by traditional Retinex filtering.

[0075] Furthermore, the reflection component of the image to be processed can be optimized by combining the results of edge detection.

[0076] This application uses a light sensor to divide the image into three regions in real time: overly bright (>5000 lux), normal (500-5000 lux), and overly dark (<500 lux). It then employs targeted strategies such as gamma compression, natural preservation (i.e., no processing), and Retinex enhancement to achieve adaptive correction across the entire illumination range (0-10000 lux). This partitioned processing makes the algorithm more targeted and highly applicable. It prevents overexposure or underexposure of the image, thus preventing the loss of image details and improving the accuracy of target recognition during target tracking, thereby increasing the success rate of visual tracking in uneven lighting conditions.

[0077] In one embodiment, optimizing the reflection component of the image to be processed includes: Edge detection is performed on the image to be processed to obtain the edge detection results; Based on the edge detection results, the reflection component of the image to be processed is optimized.

[0078] Specifically, when optimizing the reflection component of an image, since the reflection component describes the inherent properties of an object's surface (such as color, texture, and material) and is composed of high-frequency details, enhancing the reflection component can improve the local contrast and detail clarity of the image. Therefore, this application can introduce a sparse attention mechanism to assign higher weights to edge regions in the reflection component: ; in, This represents the enhanced image to be processed; This represents the reflection component of the image to be processed; The edge detection results (specifically, a binary image of 0-1) obtained by Canny edge detection of the image to be processed in advance.

[0079] This application, by introducing a sparse attention mechanism and combining it with edge detection results, can improve the detail retention rate in dark areas by 28%, effectively avoiding distortion caused by changes in illumination. This prevents the loss of image details, helps improve the accuracy of target recognition during target tracking, and thus increases the success rate of visual tracking in uneven lighting scenarios.

[0080] In one embodiment, based on the rate of illumination change at the time the image was captured, a smoothing filter is applied to the image after exposure compensation processing, including: The smoothing filter factor is determined based on the rate of change of illumination at the time of image capture. Based on the smoothing filter factor, the image to be processed after exposure compensation is subjected to smoothing filtering.

[0081] It should be noted that in scenarios with drastic changes in illumination, such as entering and exiting tunnels or flashing lights, this application can use an illumination transient smoothing module to perform a three-stage processing of the image to be processed, namely prediction, filtering, and switching, to reduce the interference of drastic fluctuations in illumination components on feature matching and reduce the tracking loss rate in scenarios with sudden changes in illumination (±5000 lux / s).

[0082] Specifically, the prediction part has been explained in the preceding sections, and will not be repeated here.

[0083] After obtaining the predicted rate of illumination change at the time of image capture, this application can determine the smoothing filter factor based on the rate of illumination change. .

[0084] Specifically, when When the rate of change of illumination changes drastically, the smoothing filter factor can be adjusted. It was determined to be the smaller value of 0.1. And... At that time, the smoothing filter factor will be... It was determined to be a relatively large value of 0.5.

[0085] Furthermore, the image to be processed after exposure compensation can be smoothed using the following formula: ; in, The image to be processed after exposure compensation. This is the smoothed and filtered image of the previous frame of the image to be processed. The image to be processed is the filtered image. This is the smoothing filter factor.

[0086] In one embodiment, this application may also use a light prediction model to output a light prediction value for the next 50 milliseconds, when the predicted light change rate... At that time, the smoothing filter process for the image to be processed is initiated, and compensation is triggered in advance.

[0087] The filtered and smoothed image obtained in this application ensures sufficiently low image noise even under drastic lighting conditions. As input to the subsequent feature fusion module, this guarantees the reliability of feature extraction. This effectively alleviates the problem of target tracking failure in scenes with drastic lighting changes.

[0088] This application addresses scenarios with sudden changes in illumination by integrating the "feedforward-feedback" concept from control theory. It achieves proactive defense through LSTM prediction, suppresses fluctuations through dynamic exponential moving average (EMA) filtering, and ensures tracking continuity through feature switching. The synergy of these three elements reduces the tracking loss rate by 85%.

[0089] In one embodiment, target tracking based on the smoothed image to be processed includes: Visual features are extracted from the image to be processed after smoothing filtering to obtain visual features; Geometric features are extracted from the image to be processed after smoothing filtering to obtain geometric features; Feature fusion weights are determined based on the rate of change of illumination. Based on feature fusion weights, visual features and geometric features are fused to obtain fused features; Target following based on fused features.

[0090] Specifically, this application can extract visual features and geometric features from the smoothed and filtered image by feature fusion, thereby obtaining visual features containing image texture and geometric features containing image depth information.

[0091] Furthermore, the feature fusion weights for this operation can be determined using the rate of change in illumination. .

[0092] Specifically, when the illumination is stable (i.e., the rate of change of illumination) )hour =0.7, with visual feature following as the primary driver and geometric features as a secondary driver. When there is a sudden change in illumination (i.e., the rate of change in illumination)... )hour =0.3, with geometric features as the primary driver and visual features as an auxiliary driver.

[0093] Then, based on the feature fusion weights, visual features and geometric features are fused. Specifically, this includes: ; in, Indicates fusion characteristics; Indicates visual features; Representing geometric features; This represents the feature fusion weights.

[0094] After obtaining the fusion weights, the fusion features can be output to the follower module.

[0095] Furthermore, the following module can perform target detection using deep learning object detection methods based on the fused features, and determine the target to be followed through feature matching, thereby achieving continuous target following. For example... Figure 2 As shown, this application can further determine the position coordinates of the target to be followed, and generate corresponding motion control commands based on the position coordinates to follow the target.

[0096] This application achieves dynamic weighted fusion of visual and geometric features of the image under processing, ensuring the usability of the fused features even when illumination changes. This effectively alleviates the problem of target tracking failure in scenes with drastic lighting variations.

[0097] Based on changes in illumination, visual and geometric features are fused in different proportions. When the illumination changes significantly, geometric features take precedence, while when the illumination changes slightly, visual features take precedence. This ensures the usability of the features and reduces the false negative rate.

[0098] Furthermore, this application, through its algorithm-based hardware design, achieves the tracking accuracy of multi-sensor solutions under pure monocular camera conditions, while simultaneously reducing cost and power consumption, thus possessing significant commercial competitiveness. On designated products (e.g., the RK3588 chip), this application can achieve an operating speed of nearly 20 frames per second (FPS), meeting the needs of low-computing-power devices. Moreover, this application can function normally even at low frame rates. Therefore, a balance can be struck between tracking real-time performance and accuracy.

[0099] The visual following device provided in this application is described below. The visual following device described below can be referred to in correspondence with the visual following method described above.

[0100] Furthermore, this application also provides a visual following device.

[0101] The visual tracking device includes: The exposure compensation module is used to perform exposure compensation processing on the image to be processed based on the real illumination information at the time of shooting. A smoothing filter module is used to perform smoothing filter processing on the image to be processed after exposure compensation based on the illumination change rate at the time of shooting; the illumination change rate is predicted before shooting the image to be processed. The target following module is used to perform target following based on the image to be processed after smoothing filtering.

[0102] The visual tracking device of this application performs exposure compensation processing on the image to be processed based on the real illumination information at the time of shooting, which can prevent the image to be processed from being overexposed or underexposed, thereby preventing the loss of image details. Furthermore, based on the illumination change rate at the time of shooting, the image to be processed after exposure compensation is smoothed and filtered. Since the illumination change rate is predicted before shooting the image to be processed, the interference of drastic illumination changes on target tracking can be reduced. Therefore, target tracking based on the image to be processed after smoothing and filtering can effectively improve the accuracy of target recognition during target tracking, thereby improving the success rate of visual tracking in uneven lighting scenes.

[0103] In one embodiment, the exposure compensation module is specifically used for: Determine the brightness range of the actual illumination information at the time the image to be processed was captured; The exposure compensation method is determined based on the brightness range; wherein, each brightness range corresponds to one exposure compensation method; the exposure compensation method is one of image compression, no processing, and image enhancement; The exposure compensation process is applied to the image to be processed based on the aforementioned exposure compensation method.

[0104] In one embodiment, the exposure compensation module is further configured to: If the exposure compensation method is image compression, then gamma compression is performed on the image to be processed; If the exposure compensation method is no processing, then no compensation will be performed on the image to be processed; If the exposure compensation method is image enhancement, then the image to be processed is optimized for both illumination and reflection components.

[0105] In one embodiment, the exposure compensation module is further configured to: Edge detection is performed on the image to be processed to obtain the edge detection result; Based on the edge detection results, the reflection component of the image to be processed is optimized.

[0106] In one embodiment, the smoothing filter module is specifically used for: The smoothing filter factor is determined based on the rate of change of illumination at the time of shooting the image to be processed. Based on the smoothing filter factor, the image to be processed after exposure compensation is subjected to smoothing filter processing.

[0107] In one embodiment, the target following module is specifically used for: Visual features are extracted from the image to be processed after smoothing filtering to obtain visual features; Geometric features are extracted from the image to be processed after smoothing filtering to obtain geometric features; The feature fusion weights are determined based on the rate of change in illumination. Based on the feature fusion weights, the visual features and the geometric features are fused to obtain fused features; Target following is performed based on the fused features.

[0108] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the following method: performing exposure compensation processing on the image to be processed based on the real illumination information at the time of image capture; Based on the illumination change rate at the time of image capture, the image to be processed after exposure compensation is subjected to smoothing filtering; the illumination change rate is predicted before the image to be processed is captured. Target tracking is performed based on the image to be processed after smoothing filtering.

[0109] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] In another aspect, embodiments of this application also provide a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the methods provided in the above embodiments, such as: performing exposure compensation processing on the image to be processed based on the real illumination information at the time of shooting the image to be processed. Based on the illumination change rate at the time of image capture, the image to be processed after exposure compensation is subjected to smoothing filtering; the illumination change rate is predicted before the image to be processed is captured. Target tracking is performed based on the image to be processed after smoothing filtering.

[0111] In another aspect, embodiments of this application also provide a computer program product, on which a computer program is stored, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments, such as: performing exposure compensation processing on the image to be processed based on the real illumination information at the shooting time of the image to be processed; Based on the illumination change rate at the time of image capture, the image to be processed after exposure compensation is subjected to smoothing filtering; the illumination change rate is predicted before the image to be processed is captured. Target tracking is performed based on the image to be processed after smoothing filtering.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate this application and are not intended to limit this application. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application.

Claims

1. A visual following method, characterized in that, include: Based on the real illumination information at the time the image to be processed was captured, exposure compensation processing is performed on the image to be processed. Based on the illumination change rate at the shooting time of the image to be processed, the image to be processed after exposure compensation is subjected to smoothing filtering. The rate of change of illumination is predicted before the image to be processed is captured. Target tracking is performed based on the image to be processed after smoothing filtering.

2. The visual following method according to claim 1, characterized in that, The target tracking based on the image to be processed after smoothing filtering includes: Visual features are extracted from the image to be processed after smoothing filtering to obtain visual features; Geometric features are extracted from the image to be processed after smoothing filtering to obtain geometric features; The feature fusion weights are determined based on the rate of change in illumination. Based on the feature fusion weights, the visual features and the geometric features are fused to obtain fused features; Target following is performed based on the fused features.

3. The visual following method according to claim 1, characterized in that, The exposure compensation processing of the image to be processed based on the real illumination information at the time of shooting includes: Determine the brightness range of the actual illumination information at the time the image to be processed was captured; The exposure compensation method is determined based on the brightness range; wherein, each brightness range corresponds to one exposure compensation method; the exposure compensation method is one of image compression, no processing, and image enhancement; The exposure compensation process is applied to the image to be processed based on the aforementioned exposure compensation method.

4. The visual tracking method according to claim 3, characterized in that, The exposure compensation processing of the image to be processed based on the exposure compensation method includes any one of the following: If the exposure compensation method is image compression, then gamma compression is performed on the image to be processed; If the exposure compensation method is no processing, then no compensation will be performed on the image to be processed; If the exposure compensation method is image enhancement, then the image to be processed is optimized for both illumination and reflection components.

5. The visual following method according to claim 4, characterized in that, Optimizing the reflection components of the image to be processed includes: Edge detection is performed on the image to be processed to obtain the edge detection result; Based on the edge detection results, the reflection component of the image to be processed is optimized.

6. The visual following method according to claim 1, characterized in that, The smoothing filtering process performed on the image to be processed after exposure compensation, based on the illumination change rate at the time of image capture, includes: The smoothing filter factor is determined based on the rate of change of illumination at the time of shooting the image to be processed. Based on the smoothing filter factor, the image to be processed after exposure compensation is subjected to smoothing filter processing.

7. A visual following device, characterized in that, include: The exposure compensation module is used to perform exposure compensation processing on the image to be processed based on the real illumination information at the time of shooting. A smoothing filter module is used to perform smoothing filter processing on the image to be processed after exposure compensation based on the illumination change rate at the time of shooting; the illumination change rate is predicted before shooting the image to be processed. The target following module is used to perform target following based on the image to be processed after smoothing filtering.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the visual following method as described in any one of claims 1 to 6.

9. A storage medium, said storage medium being a non-transitory computer-readable storage medium, wherein a computer program is stored thereon, characterized in that, When the computer program is executed by a processor, it implements the visual following method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the visual following method as described in any one of claims 1 to 6.

Images