Robotic head adaptive panoramic imaging method and system

Abstract

The embodiment of the application provides a kind of robot head self-adapting panoramic imaging method and system, belong to robot vision technical field.The method comprises: obtaining the lens parameter corresponding to current fisheye lens, and determining the corresponding panoramic mapping relationship according to the lens parameter;Based on the panoramic mapping relationship, the image collected by the fisheye lens is mapped and processed, to obtain initial panoramic image, and determine image change information according to the initial panoramic image of continuous frame and robot head posture change information;According to the image change information, the initial panoramic image is corrected and processed, to obtain stable panoramic image;According to the stable panoramic image, determine image focus area, and differentially process different focus areas, output panoramic image data.The lens adaptation capability of the robot head panoramic imaging, motion stability and data processing efficiency are improved in the application scheme.

Description

Technical Field

[0001] This invention relates to the field of robot vision technology, specifically to a robot head adaptive panoramic imaging method and system. Background Technology

[0002] As the application of intelligent robots in scenarios such as inspection, service, industrial operations, and mobile interaction continues to expand, higher demands are being placed on robots' ability to perceive their surroundings. Visual perception systems, as a crucial component for robots to acquire information about their external environment, typically use camera modules mounted on the robot's head to capture environmental images and then use this image information to perform functions such as target recognition, path planning, and interactive control. Due to the limited internal space of the robot's head, and the need to accommodate other sensors, controllers, and mechanical structures, camera modules usually need to be small in size and low in weight while meeting imaging performance requirements.

[0003] In existing robot vision systems, fisheye lenses are typically used to acquire large field-of-view images in order to expand the range of environmental perception. These fisheye images are then converted into panoramic images through image processing. However, different models of fisheye lenses differ in their imaging range, optical distortion characteristics, and installation positions. This means that when different lenses are used with the same camera module, it is difficult to maintain a consistent field of view and imaging effect in the output image, thus affecting the versatility of the robot vision system.

[0004] Meanwhile, as the robot moves, its head undergoes posture changes such as rotation and vibration, resulting in continuously acquired images containing both visual changes caused by the robot's own movement and real-world changes in the external environment. Existing image processing methods struggle to accurately distinguish between these two types of changes, easily leading to issues like jitter and shifts in panoramic images, thus reducing visual perception stability.

[0005] Furthermore, as the resolution of panoramic images continues to improve, the amount of image data also increases. If a uniform processing method is used for transmission and backend analysis of complete panoramic images, it will cause significant data transmission pressure and increase backend computing resource consumption, which is detrimental to the real-time operation of the robot system. Therefore, there is an urgent need for a robot head panoramic imaging method that can adapt to changes in different imaging components, improve the stability of panoramic images, and reduce the pressure of visual data processing and transmission. Summary of the Invention

[0006] The purpose of this invention is to provide a robot head adaptive panoramic imaging method and system, so as to at least solve the problems of insufficient lens adaptability, poor motion stability and high data transmission pressure in existing robot panoramic imaging.

[0007] To achieve the above objectives, a first aspect of the present invention provides a robot head adaptive panoramic imaging method, the method comprising: acquiring lens parameters corresponding to a current fisheye lens, and determining a corresponding panoramic mapping relationship based on the lens parameters; performing mapping processing on an image acquired by the fisheye lens based on the panoramic mapping relationship to obtain an initial panoramic image, and determining image change information based on consecutive frames of the initial panoramic image and robot head posture change information; performing correction processing on the initial panoramic image based on the image change information to obtain a stable panoramic image; determining an image region of interest based on the stable panoramic image, and performing differentiated processing on different regions of interest to output panoramic image data.

[0008] A second aspect of the present invention provides a robot head adaptive panoramic imaging system, the system comprising: an acquisition unit, configured to acquire lens parameters corresponding to a current fisheye lens, and determine a corresponding panoramic mapping relationship based on the lens parameters; a mapping unit, configured to perform mapping processing on the image acquired by the fisheye lens based on the panoramic mapping relationship to obtain an initial panoramic image, and determine image change information based on consecutive frames of the initial panoramic image and robot head posture change information; a correction unit, configured to perform correction processing on the initial panoramic image based on the image change information to obtain a stable panoramic image; and an output unit, configured to determine the image region of interest based on the stable panoramic image, perform differentiated processing on different regions of interest, and output panoramic image data.

[0009] The above technical solution achieves unified panoramic image generation under different lens conditions by acquiring fisheye lens parameters and establishing corresponding panoramic mapping relationships, thus improving the camera module's adaptability to different imaging components. By combining continuous frame image change information and robot head posture change information, image shifts caused by robot movement are corrected, improving the stability of panoramic imaging during robot movement. Simultaneously, the region of interest is determined based on the stable panoramic image and processed differentially, reducing the data processing and transmission burden on non-critical areas. This ensures the effective preservation of key visual information while improving the real-time processing capability of the robot's vision system. Through these methods, the adaptability, stability, and data processing efficiency of the robot's head panoramic imaging system are enhanced.

[0010] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0011] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1This is a flowchart of the steps of a robot head adaptive panoramic imaging method provided in one embodiment of the present invention; Figure 2 This is a flowchart of the robot head adaptive panoramic imaging process provided by one embodiment of the present invention; Figure 3 This is a schematic diagram of fisheye image to panoramic image coordinate mapping provided by one embodiment of the present invention; Figure 4 This is a system structure diagram of a robot head adaptive panoramic imaging system provided in one embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0013] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0014] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0015] Figure 1 This is a flowchart of a robot head adaptive panoramic imaging method according to one embodiment of the present invention. Figure 1 As shown, this invention provides a robot head adaptive panoramic imaging method, the method comprising: Step S10: Obtain the lens parameters corresponding to the current fisheye lens, and determine the corresponding panoramic mapping relationship based on the lens parameters.

[0016] Specifically, the process involves acquiring lens recognition information corresponding to the current fisheye lens, determining the corresponding target lens parameters from a preset set of lens parameters based on the lens recognition information, determining the imaging region parameters and projection transformation parameters corresponding to the current fisheye lens based on the target lens parameters, and determining the effective imaging region in the image acquired by the fisheye lens based on the imaging region parameters, and establishing a coordinate mapping relationship between the effective imaging region and the target panoramic coordinate system based on the projection transformation parameters to obtain the panoramic mapping relationship.

[0017] In this embodiment of the invention, lens identification information corresponding to the currently installed fisheye lens is obtained. This lens identification information is used to distinguish different models of fisheye lenses; it is stored in the identification area of ​​the lens adapter structure and is read after the fisheye lens is installed. Based on the lens identification information, the corresponding target lens parameters are matched from a preset set of lens parameters. The target lens parameters include lens model, focal length, fisheye projection type, effective imaging range parameters, and lens mounting offset parameters, etc., used to describe the imaging characteristics of the current fisheye lens.

[0018] Furthermore, the imaging region parameters and projection transformation parameters corresponding to the current fisheye lens are determined based on the target lens parameters. The imaging region parameters are used to determine the effective imaging region in the image acquired by the fisheye lens. Since the original image acquired by the fisheye lens typically includes an effective imaging region and an outer invalid region, the location range of the effective imaging region is determined based on the imaging region parameters to avoid invalid regions participating in subsequent panoramic unfolding processing. For example, for a fisheye original image with a resolution of 1920×1080, if the effective imaging region is determined to be a circular region with a radius of 800 pixels based on the image center according to the target lens parameters, then panoramic mapping is performed only on the pixel data within this circular region.

[0019] Furthermore, a coordinate mapping relationship between the effective imaging area and the target panoramic coordinate system is established based on the projection transformation parameters. The projection transformation parameters characterize the transformation relationship between fisheye image coordinates and panoramic image coordinates, including transformation parameters corresponding to the fisheye projection model, lens optical axis offset parameters, and mounting rotation parameters. Based on the projection transformation parameters, the pixel positions in the effective imaging area are mapped to the target panoramic coordinate system to obtain the panoramic mapping relationship. The target panoramic coordinate system corresponds to a preset panoramic output range, used to ensure that images acquired by different fisheye lenses have a uniform output field of view and image size after mapping processing.

[0020] By employing the above method, a corresponding panoramic mapping relationship is established based on the actual imaging parameters of the current fisheye lens. This allows different models of fisheye lenses, when installed in the same camera module, to generate subsequent panoramic images according to the corresponding mapping relationship. It should be noted that the specific composition of the lens parameters is not limited to the parameters described above; any parameters that reflect the imaging characteristics of the fisheye lens are within the scope of this solution. The specific field of view and output size of the target panoramic coordinate system are pre-set according to the robot's visual acquisition requirements.

[0021] Step S20: Based on the panoramic mapping relationship, perform mapping processing on the image acquired by the fisheye lens to obtain an initial panoramic image, and determine the image change information based on the initial panoramic image in consecutive frames and the robot head posture change information.

[0022] Specifically, the process of mapping the image acquired by the fisheye lens based on the panoramic mapping relationship to obtain an initial panoramic image includes: acquiring the original fisheye image acquired by the fisheye lens; determining the target panoramic coordinates corresponding to each effective pixel in the original fisheye image according to the panoramic mapping relationship; performing coordinate transformation on the effective pixel based on the target panoramic coordinates, mapping the effective imaging area in the original fisheye image to the target panoramic coordinate system, and obtaining the initial panoramic image.

[0023] Furthermore, based on the target panoramic coordinates, coordinate transformation is performed on the effective pixels to map the effective imaging area in the original fisheye image to the target panoramic coordinate system, thereby obtaining the initial panoramic image. This includes: determining the target panoramic coordinates corresponding to the effective pixels according to the panoramic mapping relationship, and establishing a pixel mapping grid based on the target panoramic coordinates corresponding to each effective pixel; adjusting the pixel mapping grid using non-uniform sampling according to the degree of distortion corresponding to different regions in the original fisheye image, so that the sampling density corresponding to high distortion regions is higher than the sampling density corresponding to low distortion regions; based on the adjusted pixel mapping grid, mapping the pixel values ​​in the effective imaging area to the target panoramic coordinate system, and generating the initial panoramic image according to the pixel arrangement relationship in the target panoramic coordinate system.

[0024] Furthermore, determining image change information based on the initial panoramic image and robot head pose change information in consecutive frames includes: acquiring image feature points in adjacent frames of the initial panoramic image, and determining feature point displacement information based on the position changes of the image feature points in adjacent frames; acquiring the robot head pose change information, and determining predicted displacement information corresponding to head movement based on the robot head pose change information; and matching the feature point displacement information with the predicted displacement information to determine image change information in the initial panoramic image caused by robot head movement.

[0025] In this embodiment of the invention, a raw fisheye image acquired by a fisheye lens is obtained. The raw fisheye image is the raw image data output by the image sensor based on the current imaging result of the fisheye lens, and this image data includes pixel information corresponding to the effective imaging area of ​​the fisheye lens. Since fisheye lenses typically employ a large field-of-view imaging method, the pixel distribution in the raw image differs from that of conventional perspective imaging. Therefore, it is necessary to expand the raw fisheye image based on the panoramic mapping relationship determined in step S10.

[0026] Furthermore, the target panoramic coordinates corresponding to each valid pixel in the original fisheye image are determined based on the panoramic mapping relationship. Specifically, the panoramic mapping relationship records the correspondence between pixel positions within the effective imaging area and the target panoramic coordinate system. For each valid pixel in the original fisheye image, its corresponding target panoramic coordinates are queried according to its position index in the fisheye coordinate system. For example, for a valid pixel with coordinates (u, v) in the original fisheye image, its corresponding position (X, Y) in the target panoramic coordinate system is determined according to the panoramic mapping relationship, thus establishing a mapping relationship between the pixel's coordinates from the fisheye image coordinates to the panoramic image coordinates.

[0027] Furthermore, to improve image continuity and edge region imaging quality during the fisheye lens unfolding process, a pixel mapping grid is established based on the target panoramic coordinates corresponding to each effective pixel. This pixel mapping grid describes the arrangement of effective pixels in the original fisheye image within the target panoramic coordinate system. Since fisheye lenses exhibit varying degrees of spatial distortion in different regions, with pixels near the edges typically stretched more than those in the center, the pixel mapping grid is adjusted using non-uniform sampling based on the degree of distortion in different regions of the original fisheye image.

[0028] Specifically, the distortion level of different regions is determined based on the projection transformation parameters in the target lens parameters, and multiple sampling regions are divided according to the distortion level. For edge regions with higher distortion levels, the sampling density of the pixel mapping grid is increased to reduce the spatial distance between adjacent sampling points, thereby preserving more edge image details. For central regions with lower distortion levels, the sampling density of the pixel mapping grid is decreased to reduce redundant sampling calculations while ensuring image continuity. For example, when dividing the original fisheye image into central, transition, and edge regions, the sampling interval for edge regions is set to 1 pixel, for transition regions to 2 pixels, and for central regions to 3 pixels, so that different distortion regions have matching sampling accuracy.

[0029] Based on the adjusted pixel mapping grid, pixel values ​​in the effective imaging area are mapped to the target panoramic coordinate system, and an initial panoramic image is generated according to the pixel arrangement in the target panoramic coordinate system. During the mapping process, for positions where there is no directly corresponding pixel in the target panoramic coordinate system, pixel interpolation is performed based on the positional relationship between adjacent effective pixels to ensure that the generated initial panoramic image does not have obvious hole areas. The initial panoramic image has a uniform output field of view and image size, enabling images acquired by different fisheye lenses to be converted to the same panoramic display coordinate system after processing according to the corresponding panoramic mapping relationship.

[0030] Furthermore, image change information is determined based on the initial panoramic images and robot head pose change information in consecutive frames. Specifically, image feature points in adjacent initial panoramic images are acquired, and feature point displacement information is determined based on the positional changes of these feature points in adjacent frames. The image feature points include pixel regions in the initial panoramic images with significant texture or edge changes, such as object edges, texture boundaries, and positions corresponding to fixed environmental structures. By matching the same feature point in adjacent frames, the positional changes of that feature point over continuous time are determined, yielding the corresponding feature point displacement information.

[0031] Simultaneously, robot head pose change information is acquired. This robot head pose change information characterizes the spatial pose changes of the robot head within the time interval between adjacent frame image acquisitions, including the amount of rotational change of the head around different directions. Based on the robot head pose change information, predicted displacement information corresponding to the head motion is determined. Specifically, based on the direction and amplitude of the robot head pose change, the spatial motion of the head is transformed into the panoramic coordinate system corresponding to the initial panoramic image, obtaining the theoretical changes in feature point positions caused by the robot head motion.

[0032] Furthermore, the displacement information of the feature points is matched with the predicted displacement information to determine the image changes in the initial panoramic image caused by the robot's head movement. When the actual displacement direction and displacement distance corresponding to a certain feature point are consistent with the predicted displacement information, the image change corresponding to that feature point is determined to be a change caused by the robot's head movement; when there is a significant difference between the actual displacement and the predicted displacement, the environmental change information corresponding to that feature point is retained to avoid misjudging moving objects in the real environment as image changes caused by the robot's head movement.

[0033] For example, when the robot's head turns to the left, fixed feature points such as building edges and wall textures in the background area will be displaced accordingly according to the direction of head rotation. By comparing the actual feature point displacement with the posture prediction displacement, it can be determined that the image change in this part is mainly caused by the robot's head movement. However, if the actual displacement of the feature points corresponding to moving people or vehicles is inconsistent with the predicted displacement, then the image change information caused by head movement will not be included.

[0034] The above method first utilizes the panoramic mapping relationship corresponding to the fisheye lens to achieve unified panoramic unfolding under different lens conditions. Then, it combines the initial panoramic images of consecutive frames with information on changes in the robot's head posture to identify image changes caused by the robot's own movement, providing an accurate data foundation for subsequent panoramic image stabilization correction. It should be noted that the sampling method of the pixel mapping grid, the feature point extraction method, and the method of obtaining posture change information are not limited to the specific forms described above. As long as a unified panoramic image can be generated based on the fisheye image mapping relationship, and the corresponding image change information can be determined based on continuous image changes and changes in the robot's head posture, it falls within the scope of protection of this solution.

[0035] Preferably, to further improve the adaptability of fisheye image unfolding to dynamic environments, after generating the initial panoramic image, the pixel stability in consecutive frames of the initial panoramic image is analyzed. Specifically, based on the brightness and spatial position changes of corresponding pixels in adjacent frames of the initial panoramic image, an image stability reference region is determined, and the panoramic mapping relationship is locally constrained and adjusted based on the image stability reference region.

[0036] The image stabilization reference region is used to characterize environmental structural regions that remain relatively stable over a continuous period of time, such as wall surfaces, ground textures, and the edges of fixed facilities. Based on the pixel change trend within the stabilization reference region, a consistency check is performed on the local regions after panoramic mapping. When a persistent offset is detected in a local region, the positions of local nodes in the corresponding pixel mapping grid are fine-tuned to ensure that the adjusted mapping result remains spatially consistent with the continuous frame images.

[0037] By using the above method, without changing the overall panoramic mapping relationship, the local mapping error is constrained by the stable area of ​​the environment, which reduces the local imaging drift caused by lens assembly micro-deviation, temperature change or long-term vibration during robot movement, and improves the stability of panoramic images during long-term operation.

[0038] Step S30: Correct the initial panoramic image according to the image change information to obtain a stable panoramic image.

[0039] Specifically, based on the image change information, a continuously changing region in the initial panoramic image is determined, and this continuously changing region is defined as a motion compensation region. The compensation direction and compensation amplitude corresponding to the motion compensation region are determined based on the image change information caused by the robot head movement. The pixel positions in the motion compensation region are adjusted based on the compensation direction and compensation amplitude, and the boundary between the adjusted motion compensation region and the unadjusted region is processed for continuity. Finally, the stable panoramic image is generated based on the processed initial panoramic image.

[0040] Furthermore, determining the compensation direction and compensation amplitude corresponding to the motion compensation area based on the head movement change information includes: determining the posture change direction and posture change amount of the robot head within a preset time interval based on the image change information caused by the robot head movement; mapping the motion compensation area to the target panoramic coordinate system based on the posture change direction and posture change amount, and determining the predicted displacement direction and predicted displacement distance corresponding to each position in the motion compensation area; determining the compensation direction corresponding to the motion compensation area based on the predicted displacement direction, and determining the compensation amplitude corresponding to the motion compensation area based on the predicted displacement distance.

[0041] In this embodiment of the invention, based on the image change information caused by the robot's head movement determined in step S20, the regions in the initial panoramic image affected by the robot's movement are identified. First, based on the image change information corresponding to each feature point in consecutive frames of the initial panoramic image, regions with continuous spatial positions and consistent change trends are determined, and these regions are defined as continuously changing regions. The continuously changing regions are used to characterize regions where the overall image position changes due to the robot's head rotation, swaying, or vibration.

[0042] Furthermore, the continuously changing region is defined as the motion compensation region, and the compensation direction and compensation amplitude corresponding to the motion compensation region are determined based on the image change information caused by the robot head movement. The compensation direction indicates the direction in which the pixel position within the motion compensation region needs to be adjusted, and the compensation amplitude indicates the distance of the pixel position adjustment.

[0043] Specifically, the direction and magnitude of the robot head's posture change within a preset time interval are determined based on image change information caused by the robot head's movement. This preset time interval corresponds to the acquisition time interval of adjacent initial panoramic images. For example, if the acquisition interval between two adjacent frames is 50ms, the corresponding posture change direction and magnitude are determined based on the robot head's posture changes within this 50ms time period. The posture change direction includes the rotational direction of the robot head around different spatial directions, and the posture change magnitude represents the degree of change in the corresponding direction.

[0044] Furthermore, based on the orientation change direction and the orientation change amount, the motion compensation region is mapped to the target panoramic coordinate system, and the predicted displacement direction and predicted displacement distance corresponding to each position in the motion compensation region are determined. Since the spatial projection relationship differs at different positions after the fisheye image is unfolded, the pixel displacements corresponding to different regions in the panoramic image are not entirely the same when the robot head undergoes the same orientation change. In this embodiment, the predicted displacement direction and predicted displacement distance corresponding to each position within the motion compensation region are determined according to the correspondence between each position in the target panoramic coordinate system and the robot head orientation change.

[0045] For example, when the robot's head rotates horizontally, the displacement distances of pixels located in the center and edge regions of the panoramic image are different. Therefore, the corresponding predicted displacement distance is determined according to the coordinate relationship of each position in the target panoramic coordinate system, instead of using a uniform translation amount to adjust the entire initial panoramic image. This reduces the problem of stretching or misalignment in local areas after the panoramic image is unfolded.

[0046] The compensation direction corresponding to the motion compensation region is determined based on the predicted displacement direction, and the compensation amplitude corresponding to the motion compensation region is determined based on the predicted displacement distance. After obtaining the compensation direction and compensation amplitude, the pixel positions in the motion compensation region are adjusted to restore the image content that has been offset by the robot's head movement to its corresponding position.

[0047] Furthermore, to avoid significant boundary changes between the motion-compensated area and the unadjusted area, after pixel position adjustment, the connection boundary between the adjusted motion-compensated area and the unadjusted area is processed for continuity. Specifically, based on the grayscale variation and spatial position relationship of adjacent pixels at the edge of the compensation area, the pixel values ​​in the boundary transition area are smoothly adjusted to maintain a continuous image change trend between the compensation area and the unadjusted area, reducing abrupt changes, ghosting, or edge breaks during local compensation.

[0048] A stable panoramic image is generated based on the processed initial panoramic image. The stable panoramic image retains image information corresponding to changes in the real environment and reduces non-environmental image shifts caused by robot head movements, thus providing more stable input data for subsequent target recognition or region processing based on the panoramic image.

[0049] It should be noted that the determination method of the motion compensation area, the calculation method of the compensation direction and the compensation amplitude, and the boundary continuity processing method are not limited to the specific forms mentioned above. As long as the corresponding compensation relationship can be determined based on the image change information caused by the robot head movement, and the position correction of the initial panoramic image can be performed to obtain a stable panoramic image, they all fall within the protection scope of this solution.

[0050] Preferably, the distribution of regional motion change is established based on the predicted displacement direction and predicted displacement distance corresponding to multiple feature points within the motion compensation area, and the local compensation weight is determined based on the degree of displacement continuity between adjacent positions.

[0051] When the predicted displacement direction and predicted displacement distance of multiple adjacent positions within the motion compensation area are consistent, the corresponding area is determined as a stable motion area, and overall correction is performed according to the corresponding compensation direction and compensation amplitude. When there are obvious displacement differences in a local area, the area is divided into a refined compensation area, and the compensation amount at the corresponding position is calculated separately to avoid local image stretching or ghosting caused by using uniform compensation parameters.

[0052] By using the above method, the overall image changes caused by the robot's head movement are further subdivided into different motion consistency regions, making the panoramic image correction process more consistent with the spatial change characteristics after the fisheye is unfolded. This improves the continuity and visual quality of stable panoramic images under complex motion states such as rapid head turning and walking vibration.

[0053] Step S40: Determine the region of interest based on the stable panoramic image, perform differentiated processing on different regions of interest, and output panoramic image data.

[0054] Specifically, determining the image region of interest based on the stable panoramic image includes: performing target detection on the stable panoramic image to determine the target detection region in the stable panoramic image; obtaining the robot's current motion direction and determining the motion region of interest in the panoramic coordinate system corresponding to the stable panoramic image based on the robot's current motion direction; and fusing the target detection region and the motion region of interest based on the spatial positional relationship between the target detection region and the motion region of interest to obtain the image region of interest.

[0055] Furthermore, different regions of interest are processed differently to output panoramic image data, including: determining the corresponding region processing range based on the region of interest in the image, and dividing the stable panoramic image into regions of interest and non-in-interest; determining corresponding region processing parameters based on the positional relationship between the regions of interest and non-in-interest; performing image processing on the regions of interest and non-in-interest based on the region processing parameters, and combining the processed images of different regions to obtain the panoramic image data.

[0056] In this embodiment of the invention, the region of interest is determined based on the stable panoramic image obtained in step S30. First, target detection is performed on the stable panoramic image to determine the target detection region in the stable panoramic image. The target detection is used to identify target objects in the stable panoramic image that require priority preservation of image details, and to determine the spatial location range of the target object in the panoramic coordinate system. The target detection region includes the boundary range corresponding to the target object and its position coordinates in the stable panoramic image.

[0057] For example, during robot inspection, when a person, obstacle, or equipment marker is detected in a stable panoramic image, the coverage area of ​​the corresponding target object in the panoramic image is determined based on the detection results, and this area is used as the target detection area. Areas where no target object is detected are not included in the subsequent processing.

[0058] Furthermore, the robot's current direction of movement is obtained, and the region of interest is determined in the panoramic coordinate system corresponding to the stable panoramic image based on the robot's current direction of movement. Since the region corresponding to the robot's direction of movement usually contains path information, obstacle information, and environmental change information during robot movement, it is necessary to prioritize ensuring the image quality of this region.

[0059] Specifically, the robot's current direction of movement is transformed into the panoramic coordinate system corresponding to the stable panoramic image. The spatial coverage area corresponding to this direction is determined based on a preset field of view, thus obtaining the region of interest (GI). For example, when the robot moves forward, the region of interest is defined as the area centered on the projection of the robot's current direction of movement in the panoramic coordinate system, with a horizontal field of view of 90° and a vertical field of view of 60°. The field of view is preset based on the robot's speed, sensing distance, and application scenario requirements.

[0060] Furthermore, based on the spatial relationship between the target detection region and the motion interest region, the target detection region and the motion interest region are fused to obtain the image interest region. Specifically, when the target detection region and the motion interest region spatially overlap, the overlapping regions are merged to avoid redundant processing; when the target detection region is located outside the motion interest region, the target detection region is retained and added to the key processing range outside the motion interest region, thereby ensuring that all important targets around the robot can be included in the image interest region.

[0061] After obtaining the region of interest in the image, the corresponding region processing range is determined based on the region of interest, and the stable panoramic image is divided into regions of interest and regions of non-interest. The region of interest is the pixel range corresponding to the region of interest in the image, and the region of non-interest is the remaining area in the stable panoramic image excluding the region of interest.

[0062] Furthermore, based on the positional relationship between the regions of interest and the regions of non-interest, corresponding region processing parameters are determined. These region processing parameters control the image processing methods for different regions, including encoding quality parameters, image update cycle parameters, and compression processing parameters. For example, higher encoding quality parameters and shorter update cycles are set for regions of interest, while lower encoding quality parameters and longer update cycles are set for regions of non-interest, so that limited data transmission resources are prioritized for regions where the robot's current perception needs are higher.

[0063] Image processing is performed on the region of interest and the region of non-interest based on the aforementioned region processing parameters. The processed images from different regions are then combined to obtain the panoramic image data. Specifically, image detail information in the region of interest is preserved, while a data reduction method is applied to the region of non-interest, reducing the overall data size while maintaining the integrity of the panoramic image. The processed images of the region of interest and the region of non-interest are then recombined according to their original spatial positions within the stable panoramic image to form complete panoramic image data.

[0064] By using the above method, key image areas are determined based on the robot's current motion state and environmental perception requirements, and differentiated processing is implemented for different areas, so that the output panoramic image data can reduce the data processing and transmission pressure while ensuring the integrity of key visual information.

[0065] It should be noted that the specific forms of the target detection method, the motion attention region determination method, and the region processing parameters are not limited to the above embodiments. As long as it is possible to determine image regions with different levels of attention based on a stable panoramic image and to implement differentiated image processing according to the level of attention of the regions, they all fall within the protection scope of this solution.

[0066] In a specific embodiment, such as Figure 2 As shown, this embodiment provides a robot head adaptive panoramic imaging method. The method first acquires the lens recognition information corresponding to the current fisheye lens, and then matches the corresponding target lens parameters based on the lens recognition information. Based on the target lens parameters, it determines the imaging area parameters and projection transformation parameters corresponding to the fisheye lens, establishing a panoramic mapping relationship between the fisheye image and the target panoramic coordinate system. By establishing corresponding mapping relationships for different fisheye lenses, even when different models of fisheye lenses are installed in the same robot head camera module, panoramic images can still be generated within a unified field of view.

[0067] Furthermore, the original fisheye image captured by the fisheye lens is acquired, and the target panoramic coordinates corresponding to each effective pixel in the original fisheye image are determined according to the aforementioned panoramic mapping relationship. The effective imaging area is then mapped to the target panoramic coordinate system to generate an initial panoramic image. Specifically, during the fisheye image unfolding process, a pixel mapping grid is established based on the degree of distortion corresponding to different regions of the fisheye lens, and the sampling density is increased for high-distortion regions to retain more image details in the edge regions and reduce image distortion caused by spatial stretching during fisheye unfolding. Figure 3 As shown, the fisheye image coordinates are transformed and mapped to the panoramic image coordinate system after coordinate transformation and mapping. At the same time, the mapping grid is adjusted according to the non-uniform sampling method so that different regions in the fisheye image can be fully unfolded according to the corresponding precision.

[0068] Furthermore, during the continuous acquisition of initial panoramic images, image feature points in adjacent frames of the initial panoramic images are obtained, and the displacement information of the feature points is determined based on their positional changes in consecutive frames. Simultaneously, robot head posture change information is acquired, and the corresponding predicted displacement information is determined based on the direction and magnitude of the robot head posture change. By matching the actual feature point displacement information with the predicted displacement information, it is determined whether the current image change is caused by robot head movement. When the image change matches the robot head posture change, the corresponding region is identified as the motion change region, and the compensation direction and magnitude are further determined; when the image change does not match the head motion change, the corresponding information is retained as environmental change information.

[0069] Furthermore, a motion compensation region is determined based on image change information caused by robot head movement, and the pixel positions within the motion compensation region are adjusted according to the compensation direction and amplitude to correct the panoramic image shift caused by robot head rotation or vibration. After pixel position adjustment, continuity processing is applied to the boundary between the motion compensation region and the unadjusted region to reduce image abrupt changes caused by local compensation, resulting in a stable panoramic image.

[0070] Furthermore, the image region of interest is determined based on the stabilized panoramic image. Specifically, target detection is performed on the stabilized panoramic image to determine the target detection region. Simultaneously, the corresponding motion region of interest is determined by combining this with the robot's current motion direction. The two regions are then fused based on their spatial relationship to obtain the final image region of interest. For different regions of importance, corresponding region processing parameters are determined, with higher processing priority for regions of interest and lower processing priority for non-regions of interest. The processed images of different regions are then recombined to generate panoramic image data.

[0071] Through the above implementation methods, by utilizing lens parameter adaptation, fisheye image mapping, motion change recognition, and regional differentiation processing, stable panoramic imaging of the robot's head camera module under different lens conditions and motion states can be achieved, thereby improving the consistency and real-time performance of panoramic visual output.

[0072] Figure 4 This is a system structure diagram of a robot head adaptive panoramic imaging system provided in one embodiment of the present invention. Figure 4 As shown, this invention provides a robot head adaptive panoramic imaging system. The system includes: an acquisition unit, used to acquire lens parameters corresponding to the current fisheye lens and determine a corresponding panoramic mapping relationship based on the lens parameters; a mapping unit, used to perform mapping processing on the image acquired by the fisheye lens based on the panoramic mapping relationship to obtain an initial panoramic image, and to determine image change information based on consecutive frames of the initial panoramic image and robot head posture change information; a correction unit, used to perform correction processing on the initial panoramic image based on the image change information to obtain a stable panoramic image; and an output unit, used to determine the image interest region based on the stable panoramic image, perform differentiated processing on different interest regions, and output panoramic image data.

[0073] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0074] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details described above. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe the various possible combinations.

[0075] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the embodiments of the present invention, they should also be regarded as the content disclosed by the embodiments of the present invention.

Claims

1. A method for adaptive panoramic imaging of a robot's head, characterized in that, The method includes: Obtain the lens parameters corresponding to the current fisheye lens, and determine the corresponding panoramic mapping relationship based on the lens parameters; The images acquired by the fisheye lens are mapped based on the panoramic mapping relationship to obtain an initial panoramic image, and the image change information is determined based on the initial panoramic image and the robot head posture change information in consecutive frames. The initial panoramic image is corrected based on the image change information to obtain a stable panoramic image; Based on the stable panoramic image, the region of interest is determined, and different regions of interest are processed differently to output panoramic image data.

2. The robot head adaptive panoramic imaging method according to claim 1, characterized in that, Obtain the lens parameters corresponding to the current fisheye lens, and determine the corresponding panoramic mapping relationship based on the lens parameters, including: Obtain the lens recognition information corresponding to the current fisheye lens, and determine the corresponding target lens parameters from the preset lens parameter set based on the lens recognition information; Based on the target lens parameters, determine the imaging region parameters and projection transformation parameters corresponding to the current fisheye lens, and determine the effective imaging region in the image acquired by the fisheye lens according to the imaging region parameters; Based on the projection transformation parameters, a coordinate mapping relationship is established between the effective imaging area and the target panoramic coordinate system to obtain the panoramic mapping relationship.

3. The robot head adaptive panoramic imaging method according to claim 1, characterized in that, Based on the aforementioned panoramic mapping relationship, the image acquired by the fisheye lens is mapped to obtain an initial panoramic image, including: Obtain the original fisheye image captured by the fisheye lens; The target panoramic coordinates corresponding to each effective pixel in the original fisheye image are determined based on the panoramic mapping relationship. Based on the target panoramic coordinates, the effective pixels are transformed to map the effective imaging area in the original fisheye image to the target panoramic coordinate system, thus obtaining the initial panoramic image.

4. The robot head adaptive panoramic imaging method according to claim 3, characterized in that, Based on the target panoramic coordinates, coordinate transformation is performed on the effective pixels to map the effective imaging area in the original fisheye image to the target panoramic coordinate system, thereby obtaining the initial panoramic image, including: The target panoramic coordinates corresponding to the effective pixels are determined according to the panoramic mapping relationship, and a pixel mapping grid is established according to the target panoramic coordinates corresponding to each effective pixel. Based on the degree of distortion in different regions of the original fisheye image, the pixel mapping grid is adjusted by non-uniform sampling so that the sampling density corresponding to the high distortion region is higher than the sampling density corresponding to the low distortion region. Based on the adjusted pixel mapping grid, the pixel values ​​in the effective imaging area are mapped to the target panoramic coordinate system, and the initial panoramic image is generated according to the pixel arrangement relationship in the target panoramic coordinate system.

5. The robot head adaptive panoramic imaging method according to claim 3, characterized in that, Image change information is determined based on the initial panoramic image and robot head pose change information in consecutive frames, including: Image feature points in the initial panoramic image of adjacent frames are obtained, and the displacement information of the feature points is determined based on the position changes of the image feature points in adjacent frames; The robot head posture change information is obtained, and the predicted displacement information corresponding to the head movement is determined based on the robot head posture change information. The feature point displacement information is matched with the predicted displacement information to determine the image change information caused by the robot head movement in the initial panoramic image.

6. The robot head adaptive panoramic imaging method according to claim 1, characterized in that, The initial panoramic image is corrected based on the image change information to obtain a stable panoramic image, including: The continuous change region in the initial panoramic image is determined based on the image change information, and the continuous change region is determined as the motion compensation region. The compensation direction and compensation amplitude corresponding to the motion compensation region are determined based on the image change information caused by the robot head movement. The pixel positions in the motion compensation region are adjusted based on the compensation direction and the compensation amplitude, and the boundary between the adjusted motion compensation region and the unadjusted region is processed for continuity. The stable panoramic image is generated based on the processed initial panoramic image.

7. The robot head adaptive panoramic imaging method according to claim 6, characterized in that, Based on the head movement change information, the compensation direction and compensation amplitude corresponding to the motion compensation area are determined, including: The direction and amount of posture change of the robot head within a preset time interval are determined based on the image change information caused by the robot head movement. Based on the attitude change direction and the attitude change amount, the motion compensation area is mapped to the target panoramic coordinate system to determine the predicted displacement direction and predicted displacement distance corresponding to each position in the motion compensation area; The compensation direction corresponding to the motion compensation region is determined based on the predicted displacement direction, and the compensation amplitude corresponding to the motion compensation region is determined based on the predicted displacement distance.

8. The robot head adaptive panoramic imaging method according to claim 1, characterized in that, Determining the region of interest based on the stabilized panoramic image includes: Target detection is performed on the stable panoramic image to determine the target detection region in the stable panoramic image; Obtain the robot's current direction of motion, and determine the region of interest in motion in the panoramic coordinate system corresponding to the stable panoramic image based on the robot's current direction of motion. Based on the spatial relationship between the target detection region and the motion attention region, the target detection region and the motion attention region are fused to obtain the image attention region.

9. The robot head adaptive panoramic imaging method according to claim 8, characterized in that, Different regions of interest are processed differently to output panoramic image data, including: The corresponding region processing range is determined based on the region of interest in the image, and the stable panoramic image is divided into regions of interest and non-regions of interest. Based on the positional relationship between the regions of interest and the regions of non-interest, the corresponding region processing parameters are determined respectively; Based on the region processing parameters, image processing is performed on the region of interest and the region of non-interest respectively, and the processed images of different regions are combined to obtain the panoramic image data.

10. A robot head adaptive panoramic imaging system, characterized in that, The system includes: The acquisition unit is used to acquire the lens parameters corresponding to the current fisheye lens and determine the corresponding panoramic mapping relationship based on the lens parameters; The mapping unit is used to perform mapping processing on the image acquired by the fisheye lens based on the panoramic mapping relationship to obtain an initial panoramic image, and to determine image change information based on the initial panoramic image in consecutive frames and the robot head posture change information. The correction unit is used to correct the initial panoramic image according to the image change information to obtain a stable panoramic image; The output unit is used to determine the region of interest in the image based on the stable panoramic image, perform differentiated processing on different regions of interest, and output panoramic image data.

Images