Target positioning method, device and system
By using camera linkage and dynamic sliding window virtual ring sweep technology in different scanning areas in the intelligent lifesaving system on water, the problems of limited vision and long ring sweep cycle of ball machines and bolts are solved, efficient target positioning and monitoring are achieved, and improved Rescue efficiency.
Patent Information
- Application Number
- CN202411058643.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-26
AI Technical Summary
In the intelligent water life-saving system, the field of vision of the ball machine and the bolt is limited and cannot fully cover the monitoring area, resulting in missed target detection and high cost; the ball machine ring sweep cycle is long, making it difficult to efficiently locate the target.
Two cameras with different scanning areas are used to process the image through dynamic sliding window virtual ring scanning technology, identify the existence status of the target, and adjust the camera based on monocular visual positioning to obtain the real-time position of the target to achieve accurate tracking of the target. and monitoring.
It effectively expands the target monitoring range, improves the target detection speed, reduces the possibility of target missed detection, and maintains the accuracy of target recognition and improves rescue efficiency.
Smart Images

Figure CN119031233A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of target positioning technology, and in particular to a target positioning method, device and system. Background Art
[0002] In the field of monitoring, dome cameras (intelligent dome cameras) and gun cameras (gun cameras) are two common monitoring devices. In the intelligent life-saving system on water, dome cameras or gun cameras are usually used to detect and locate targets. However, for gun cameras, their field of view is small and the field of view angle is fixed after installation, which cannot fully cover the monitoring area. Although the area can be covered by multiple gun cameras, the cost is high. For dome cameras, the entire monitoring area can be covered by pan-tilt scanning, but there is a problem that the larger the area, the longer the scanning cycle, which makes it easy to miss targets. Therefore, there are limitations in the application of dome cameras and gun camera scanning in the intelligent life-saving system on water, and they cannot efficiently locate targets. Summary of the invention
[0003] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a method, device and system for target positioning, which can improve the efficiency of target positioning.
[0004] According to one aspect of the present application, a target positioning method is provided, including: controlling a first camera to shoot within a first preset range to obtain a first image; controlling a second camera to shoot within a second preset range to obtain a second image; wherein the shooting area of the first camera is larger than the shooting area of the second camera; processing the first image and the second image by means of a dynamic sliding window virtual circular scanning technology to identify a target existence state; when the target existence state indicates that a target exists in the first image and no target exists in the second image, adjusting the second camera until a target exists in the second image; and obtaining the real-time position of the target based on the installation position of the second camera and a monocular vision positioning method.
[0005] In one embodiment, the target positioning method further includes: when the target existence status indicates that the target exists in the second image and the target does not exist in the first image, based on the real-time position of the target, adjusting the second camera to track the target so as to place the target in the center of the shot by the second camera; or when the target existence status indicates that the target exists in the second image and the target exists in the first image, based on the real-time position of the target, adjusting the second camera to track the target so as to place the target in the center of the shot by the second camera.
[0006] In one embodiment, the first preset range includes a scene area within a first distance from the first camera, and the second preset range includes a scene area outside the first distance and within a second distance; or the first preset range includes the entire area within the scene, and the second preset range includes a fixed scene area.
[0007] In one embodiment, the first image and the second image are processed by dynamic sliding window virtual circular scanning technology to identify the existence status of the target, including: dividing the first image or the second image into multiple area images, and marking the multiple area images with serial numbers according to the segmentation order; wherein the segmentation size of the area image is positively correlated with the distance of the target in the area image close to the camera; placing the multiple area images into an object detection model for inference to obtain target position information of the multiple area images; based on the target position information of the multiple area images and the serial numbers, restoring to the first image or the second image to obtain the existence status of the target.
[0008] In one embodiment, multiple images of the area are placed in an object detection model for inference to obtain target position information of the multiple images of the area, including: adding an attention mechanism according to the probability of target appearance; based on the attention mechanism, performing frame skipping recognition on the area images in which the target has not appeared within a first preset period; based on the attention mechanism, performing frame-by-frame recognition on the area images in which the target has appeared a preset number of times within a second preset period.
[0009] In one embodiment, the first image or the second image is divided into multiple regional images, and the multiple regional images are marked with serial numbers according to the division order, including: after the first image or the second image is divided into multiple regional images, if the target is simultaneously located in at least two adjacent regional images, the first image or the second image containing the target is expanded so that the target is located on the same regional image.
[0010] In one embodiment, the first image or the second image containing the target is expanded, including: filling the lower right boundary of the first image or the second image to an integer multiple of the sliding window prediction window size; based on a preset model recognition size, performing edge expansion on the filled first image or the second image; wherein the edge expansion means filling the four sides of the first image or the second image so that the size of the first image or the second image reaches the preset model recognition size, and the preset model recognition size represents the image size input to the object detection model for recognition.
[0011] In one embodiment, based on the installation position of the second camera and the monocular vision positioning method, the real-time position of the target is obtained, including: based on the installation position of the second camera, correcting the posture of the second camera in the initial state, and obtaining the height of the second camera from the water surface; obtaining the azimuth and pitch angles of the pan-tilt rotation of the second camera when the target is photographed; based on the azimuth and pitch angles of the pan-tilt rotation, converting the coordinates of the target point to the initial state of the first camera or the second camera; calculating the azimuth and pitch angles of the target in the reference coordinate system; based on the height of the second camera from the water surface, and the azimuth and pitch angles of the target in the reference coordinate system, obtaining the coordinates of the target on the horizontal plane.
[0012] According to another aspect of the present application, a target positioning device is provided, including: a first control module, used to control a first camera to shoot within a first preset range to obtain a first image; a second control module, used to control a second camera to shoot within a second preset range to obtain a second image; wherein the shooting area of the first camera is larger than the shooting area of the second camera; an identification module, used to process the first image and the second image through a dynamic sliding window virtual circular scanning technology to identify the existence status of the target; an adjustment module, used to adjust the second camera until the target exists in the second image when the target existence status indicates that the target exists in the first image and the target does not exist in the second image; a positioning module, used to obtain the real-time position of the target based on the installation position of the second camera and the monocular vision positioning method.
[0013] According to another aspect of the present application, a target positioning system is provided, including: a first camera, the first camera is used to shoot within a first preset range to obtain a first image; a second camera, the second camera is used to shoot within a second preset range to obtain a second image; wherein the shooting area of the first camera is larger than the shooting area of the second camera; and a positioning device, the positioning device is communicatively connected to the first camera and the second camera, and is used to execute the target positioning method described in any of the above embodiments.
[0014] The target positioning method, device and system provided by the present application can effectively expand the target monitoring range and improve the target detection speed by linking the first camera with a large shooting range and the second camera. In addition, the dynamic sliding window virtual circular scanning technology can solve the problem that the target is difficult to identify due to the wide shooting range of the first camera, maintain the accuracy of target identification, reduce the possibility of target missed detection while not reducing the accuracy of target identification, reduce information processing time through monocular vision positioning, and further improve target positioning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0016] Figure 1 It is a flowchart of a target positioning method provided by an exemplary embodiment of the present application.
[0017] Figure 2 It is a schematic diagram of the principle of the first linkage mechanism provided by an exemplary embodiment of the present application.
[0018] Figure 3 It is a schematic diagram of the principle of the second linkage mechanism provided by an exemplary embodiment of the present application.
[0019] Figure 4 It is a schematic diagram of image segmentation provided by an exemplary embodiment of the present application.
[0020] Figure 5 It is a schematic diagram of installing a second camera and discovering a target provided by an exemplary embodiment of the present application.
[0021] Figure 6 It is a schematic diagram of the principle of an imaging model of a second camera provided by an exemplary embodiment of the present application.
[0022] Figure 7 It is a schematic diagram of the structure of a target positioning device provided by an exemplary embodiment of the present application.
[0023] Figure 8 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0024] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0025] Application Overview
[0026] Globally, drowning has always been one of the important causes of accidental injury and death. Therefore, it is urgent to take effective measures to reduce drowning incidents. The Intelligent Life-saving System came into being. The Intelligent Life-saving System integrates dynamic monitoring, early warning, emergency evacuation, escape guidance, personnel search and rescue as the main way to prevent drowning and rescue people who fall into the water. The target detection technology and target positioning technology, which are the "eyes" of the intelligent life-saving system on the water, are of great importance. The quality of their performance is directly related to the rescue efficiency of the intelligent life-saving system on the water.
[0027] As one of the core issues in the field of computer vision, target detection is mainly to find targets in images and determine their categories and locations. However, mature target detection technologies are all applied in ball or gun camera environments. The scanning area and scanning time of ball or gun cameras are limited. It is more difficult to apply target detection in water areas under panoramic cameras. For large-resolution images, the targets become very small, resulting in a decrease in the speed and accuracy of target detection.
[0028] The present application provides a target positioning method, device and system, which can use two cameras with different scanning areas to work together. Compared with traditional ball cameras, it can effectively replace the mechanical circular scanning mechanism, thereby maximizing the speed of identifying people who fall into the water, reducing target omissions while not reducing the accuracy of target identification. Compared with gun cameras, it can effectively expand the target monitoring range, improve the target detection speed, and ensure rescue efficiency.
[0029] Exemplary Systems
[0030] The target positioning system may include: a first camera, the first camera is used to shoot within a first preset range to obtain a first image; a second camera, the second camera is used to shoot within a second preset range to obtain a second image; wherein the shooting area of the first camera is larger than the shooting area of the second camera; a positioning device, the positioning device is communicatively connected to the first camera and the second camera, and is used to execute the target positioning method provided in this application.
[0031] The shooting area of the first camera is larger than that of the second camera, so that the monitoring area can be expanded, the detection speed can be improved, and the rescue efficiency can be ensured. For example, the first camera uses a panoramic camera to shoot a panoramic picture of the water surface, while the second camera uses a ball camera (ball camera or spherical camera), which can rotate and tilt 360° in all directions to capture images at different angles. Although the panoramic camera can capture panoramic images, the target is small in the panoramic image and it is difficult to accurately locate it. Therefore, the panoramic camera can be linked with the ball camera (such as the ball camera that comes with the panoramic camera). When the panoramic camera captures the target and the ball camera has not captured the target, the target position is located, and the shooting angle of the ball camera is adjusted so that the ball camera focuses on the target. Based on the zoom function of the ball camera, the target can be tracked and focused to obtain a more accurate position of the target. It can be understood that the first camera can also be other cameras with a wide shooting area, not limited to the panoramic camera in the above example, and the second camera can also be other cameras that can zoom, not limited to the ball camera in the above example.
[0032] The positioning device is used to control the first camera and the second camera so as to accurately track the target, locate and track the target, and calculate the target position in real time. The target position can be provided to the intelligent life-saving robot to guide the intelligent life-saving robot to approach the target, or the pixel coordinates of the target in the image and the offset of the image center point can be calculated in real time to fine-tune the PTZ (pitch, tail, zoom) of the ball camera, thereby ensuring that the target is always in the center of the image to achieve the tracking effect of the target.
[0033] Exemplary Methods
[0034] Figure 1 is a flow chart of a target positioning method provided by an exemplary embodiment of the present application, such as Figure 1 As shown, the target positioning method includes:
[0035] S1: Control the first camera to shoot within a first preset range to obtain a first image.
[0036] The first camera is used to shoot within a first range, and the first image may be a panoramic image or an image covering a large area. For example, in the rescue field, the first image is used to capture an image of the entire rescue area, or to capture an image of a large rescue area, so that when a target appears in the rescue area, the existence of the target can be quickly identified from the first image.
[0037] S2: Control the second camera to shoot within a second preset range to obtain a second image.
[0038] The shooting area of the first camera is larger than the shooting area of the second camera.
[0039] The second camera has a certain degree of flexibility compared to the first camera. The first camera is mainly used to capture panoramic images, while the second camera is mainly used to observe a certain range of fields of view. This means that to capture multiple angles of the same object, the second camera needs to be rotated or preset. This also means that in a large area, it takes a long time for the second camera to gradually scan the target. However, after the first camera and the second camera are linked, the target can be roughly identified through the first image to determine whether the target exists, and then the second camera can provide more accurate data.
[0040] For example, the first camera is a panoramic camera, and the second camera is a dome camera. Since the panoramic camera has a 360-degree all-round view, it can cover any angle and capture all scenes around the object at once without blind spots. Therefore, the area scanned by the first camera usually does not need to be changed or moved, and it can find out whether there is a target in the entire area at the first time. The second camera is a dome camera, which can support 360° continuous rotation horizontally (a few only support 340° back and forth rotation) and 90° rotation vertically, and has a zoom function, so the second camera can scan around or fix the view. When the first camera determines that there is a target, the second camera is moved to track the target.
[0041] The linkage between the first camera and the second camera can improve the efficiency of rescue. For example, when the presence of a target is detected in the first camera, the rescue equipment is immediately dispatched and the rescue information is reported. During this process, the target is tracked and located by the second camera, and accurate target information is sent to the rescue equipment, so that rescue can be carried out in time. Compared with single ball camera and gun camera monitoring, it has the ability to identify targets faster and issue warnings faster.
[0042] It can be understood that if the first camera and the second camera are responsible for scanning two non-overlapping areas respectively, when the second camera finds the target first, the second camera will be used as the main camera to directly identify and locate the target. If the first camera finds the target first, the second camera can be mobilized to the target position for secondary detection and tracking of the target.
[0043] S3: Process the first image and the second image through the dynamic sliding window virtual ring scanning technology to identify the existence status of the target.
[0044] The resolution of images used in current deep learning is mostly 1080p (1920*1080), while if the first camera uses a panoramic camera or other cameras that can capture a wide range, the resolution will be 4K, 8K or even larger. For example, the resolution of a panoramic camera is 5520*2700. When the weights obtained by training 1080p images are directly used for YOLOv5 reasoning, the image is resized to 640*640 in the preprocessing stage, and a large amount of blank areas are filled around it, causing the target in the image to become very small and difficult to identify. Therefore, it should be considered to divide the large-resolution first image into several small images, and perform target recognition on each small image frame by frame through dynamic sliding windows. Then, the target position information of the 14 small images obtained by reasoning is restored to the original image according to the segmentation sequence number, so as to obtain the reasoning result of the entire panorama. The dynamic sliding window virtual ring scanning technology can process images with larger resolutions and improve the accuracy of identifying targets from large resolutions. Without modifying or increasing the data set, the dynamic sliding window virtual ring scanning technology is used to solve the problem of poor recognition effect of YOLOv5 in the case of panoramic large resolution. If the resolution of the second image is large, dynamic sliding window virtual ring scanning technology can also be used for processing. However, if the resolution of the second image is normal, conventional image processing technology can also be used directly for target recognition. During the monitoring process of the first camera and the second camera, the panoramic video stream and the ball camera video stream are respectively connected to the edge computing platform for target detection, and the inference results are pushed to the display control end in the form of video stream for display.
[0045] S4: When the target existence state indicates that the target exists in the first image and does not exist in the second image, adjust the second camera until the target exists in the second image.
[0046] At the same time, when a target appears in the first image but not in the second image, the second camera needs to be adjusted to track the target so that the target is included in the picture captured by the second camera. The second camera has a zoom function, which can further identify the state of the target and determine whether the target is a real target or whether the target actually exists. For example, in a rescue scenario, the first camera identifies a target suspected of being a person falling into the water and reports it to the control end. The control end controls the second camera to track and zoom in on the suspected person falling into the water based on the information and the position of the target suspected of being a person falling into the water, and further identifies whether it is a person falling into the water. If it is determined to be a person falling into the water, the accurate position information of the person falling into the water is calculated and sent to the rescue robot, so that the rescue robot can go to rescue according to the accurate position information. Alternatively, the first image can directly determine the presence of a person falling into the water through dynamic sliding window virtual circular scanning technology, but the position of the person falling into the water may be constantly changing. Since the range of the first image and the first camera is too wide, the internal and external parameters are difficult to calibrate, and therefore the calculation of the target position is more complicated and time-consuming. The second camera can be directly adjusted to track the person falling into the water, and the position of the person falling into the water can be calculated in real time. Accurate and timely rescue coordinates can be provided to the rescue robot to make up for the defects of the first camera and ensure rescue efficiency.
[0047] S5: Based on the installation position of the second camera and the monocular vision positioning method, the real-time position of the target is obtained.
[0048] Since the second camera can rotate and zoom, the real-time position of the target can be calculated by the installation position of the second camera and the pan / tilt changes generated when tracking the target, combined with the monocular vision positioning method, and sent to the intelligent life-saving robot to guide the intelligent life-saving robot to approach the target. In addition, for the tracking of the target, the PTZ (pitch, tail, zoom) of the second camera is fine-tuned by calculating the offset between the pixel coordinates of the target in the second image and the center point of the image in real time, so as to ensure that the target is always in the center of the second image and achieve the tracking effect of the target. The combination of the first camera and the second camera can not only make up for each other's shortcomings, but also effectively improve the time of identification and positioning, and improve the overall rescue efficiency.
[0049] In one embodiment, the target positioning method may further include: when the target existence state indicates that the target exists in the second image and the target does not exist in the first image, adjusting the second camera to track the target based on the real-time position of the target so as to place the target in the center of the second camera's shooting; or when the target existence state indicates that the target exists in the second image and the target exists in the first image, adjusting the second camera to track the target based on the real-time position of the target so as to place the target in the center of the second camera's shooting.
[0050] The first camera and the second camera may scan the same scene, or they may scan two different areas in the entire scene. When the first camera finds the target first, the second camera is moved to the target position to detect the target. If the second camera finds the target first, the first camera may not be operated. The first camera only provides the function of expanding the monitoring range. The location of the specific target is calculated through the relevant information of the second camera. Therefore, the linkage between the first camera and the second camera can improve the overall efficiency of target recognition and target tracking.
[0051] In one embodiment, the first preset range may include a scene area within a first distance from the first camera, and the second preset range may include a scene area outside the first distance and within the second distance; or the first preset range may include the entire area within the scene, and the second preset range may include a fixed scene area.
[0052] In the first linkage mechanism (the first preset range includes the entire area within the scene, and the second preset range includes a fixed scene area), the first camera and the second camera can scan the same scene. If the scene area is small, the first camera directly captures a panoramic view of the scene, while the shooting angle of the second camera remains fixed. Only when the first camera finds the target, the second camera is controlled to move its field of view to the target position and perform target detection on the image of the area. If the second camera finds that there is a target in the area, the target position is calculated and tracked.
[0053] Figure 2 is a schematic diagram of the principle of the first linkage mechanism provided by an exemplary embodiment of the present application, such as Figure 2 As shown, the panoramic camera shoots a panoramic video stream (S11), performs target detection on the panoramic video stream (S12), and pushes the stream to the display and control end (S13). If the target is detected (S14), the ball camera is controlled to move to the target position (S15). The ball camera shoots the ball camera video stream at the target position (S16), performs target detection on the ball camera video stream (S17), and determines whether the target is found in the ball camera video stream (S18). If the target is found in the ball camera video stream, the target position is calculated and tracked (S19), and then pushed to the display and control end (S20). If the target is not found in the ball camera video stream, it is also pushed to the display and control end (S20). The display and control end (or display and control device, display and control terminal) is a device used to control and manage the display system, usually composed of hardware and software.
[0054] Streaming refers to the process of transmitting the packaged content (audio and video data) in the acquisition stage to the server. In this process, it usually involves transmitting the live video signal to the network through a certain protocol (such as RTMP, HLS, etc.), and finally reaching the display and control end for display and control.
[0055] In the second linkage mechanism (the first preset range includes the scene area within a first distance from the first camera, and the second preset range includes the scene area outside the first distance and within the second distance), the function of the first camera can be further utilized. The first camera and the second camera are installed at the same position. The first camera mainly performs target detection on the area closer to itself in the scene, and the second camera mainly performs target detection on the area farther from itself in the scene. If the first camera finds the target, the second camera stops scanning, moves the second camera from the position where the scanning stops to the position of the target, and performs target detection on the second image in the field of view of the second camera at this time. If the target is found, the position of the target in the area is calculated and tracked. If no target is found, the second camera continues its own scanning and target detection process.
[0056] Figure 3 is a schematic diagram of the principle of a second linkage mechanism provided by an exemplary embodiment of the present application, such as Figure 3 As shown, the edge computing platform normally obtains the video stream for target detection. The panoramic camera contains a ball camera, so the panoramic camera and the ball camera are located at the same installation position. The panoramic camera mainly detects targets within 100m from the panoramic camera. Since the ball camera has a zoom function, it configures a reasonable focal length and a circular scanning path, so that the ball camera is mainly responsible for detecting targets within a distance of 100m to 200m in the scene. The panoramic video stream within 100m is panoramically photographed (S21), and the target is detected on the panoramic video stream (S22), and the stream is pushed to the display and control end (S23). If a target is found in the panoramic video stream (S24), the ball camera is controlled to move to the target position (S25), and the video stream captured by the ball camera is detected (S26). It is determined whether the target is found (S27). If the target is found, the target position is calculated and tracked (S28), and then pushed to the display and control end (S29). If no target is found, the camera will continue its own scanning and target detection process, that is, the camera will scan the target within a distance of 100m to 200m (S30), shoot the camera video stream (S31), and then continue to execute S26-S29. In addition, when the panoramic view executes S21, the camera will also execute S30 synchronously. If the panoramic view does not find the target, but the camera finds the target first, it will directly execute S26-S29.
[0057] In one embodiment, S3 (processing the first image and the second image through dynamic sliding window virtual circular scanning technology to identify the existence status of the target) includes: dividing the first image or the second image into multiple area images, and marking the multiple area images with serial numbers according to the segmentation order; wherein the segmentation size of the area image is positively correlated with the distance of the target in the area image close to the camera; placing the multiple area images into the object detection model for inference to obtain the target position information of the multiple area images; based on the target position information and serial numbers of the multiple area images, restoring them to the first image or the second image to obtain the target existence status.
[0058] For example, Figure 4 : is a schematic diagram of image segmentation provided by an exemplary embodiment of the present application. If the resolution of the first image or the second image is 5520*2700, and the resolution of the images used for deep learning is mostly 1080p (1920*1080), the first image or the second image with a resolution of 5520*2700 can be segmented into the following: Figure 4 In the 14 small images shown, the target close to the camera is larger, so the segmentation is larger, and the target far from the camera is smaller, so the segmentation is finer. After the segmentation is completed, the 14 images are put into the object detection model according to the segmentation sequence and the batch_size of the model is set to 14 for reasoning. Then, the target position information of the 14 small images obtained by reasoning is restored to the original image according to the segmentation sequence, so as to obtain the reasoning result of the whole panorama. If the target is recognized in any small image, it can be determined that the target exists, and the position of the target can also be inferred through the relationship between the small image where the target exists and the original image.
[0059] In one embodiment, multiple area images are placed in an object detection model for inference to obtain target position information of the multiple area images, including: adding an attention mechanism according to the probability of target appearance; based on the attention mechanism, performing frame skipping recognition on area images in which the target has not appeared within a first preset period; based on the attention mechanism, performing frame-by-frame recognition on area images in which the target has appeared a preset number of times within a second preset period.
[0060] For example, in the process of YOLOv5 recognizing small images, a simple attention mechanism can be added according to the probability of target appearance to Figure 4 Taking the segmentation scheme shown as an example, for areas where there is no target for a long time, such as 13 and 14, frame skipping recognition is performed, and for areas where targets appear for a long time, frame-by-frame recognition is performed, thereby reducing the computing power overhead of the edge computing platform.
[0061] In one embodiment, the first image or the second image is divided into multiple regional images, and the multiple regional images are marked with serial numbers according to the division order, including: after the first image or the second image is divided into multiple regional images, if the target is simultaneously located in at least two adjacent regional images, the first image or the second image containing the target is expanded so that the target is located on the same regional image.
[0062] In actual segmentation, if the target is located on the segmentation line, the target may be segmented into two or more regional images. In this case, the target will be repeatedly identified in multiple regional images. When restoring, there will be repeated frames. Therefore, in order to avoid segmenting the same target, a certain amount of expansion will be performed between two adjacent images. In addition, when restoring, there will be repeated frames. In order to remove the repeated frames, the target information is restored to the original image and non-maximum suppression (NMS) is performed once, which can effectively solve this problem.
[0063] In one embodiment, a first image or a second image containing a target is expanded, including: filling the lower right border of the first image or the second image to an integer multiple of the sliding window prediction window size; based on a preset model recognition size, performing edge expansion on the filled first image or the second image; wherein the edge expansion means filling the four sides of the first image or the second image so that the size of the first image or the second image reaches the preset model recognition size, and the preset model recognition size represents the image size input to the object detection model for recognition.
[0064] The segmentation method can also be used as follows: 1. Pixel filling is performed first, for example, filling the lower right boundary to an integer multiple of the sliding window prediction window size to facilitate integer cutting; the width of the original image is divided by the patchsize to obtain a remainder, and the remainder just calculated is subtracted from the patchsize. The difference is the number of pixels that need to be filled in the horizontal (width) direction. At this time, the difference pixels are filled in the rightmost side of the original image. Similarly, the height of the original image is divided by the patchsize to obtain a remainder, and the remainder just calculated is subtracted from the patchsize. The difference is the number of pixels that need to be filled in the vertical (height) direction. At this time, the difference pixels are filled in the bottom of the original image. 2. Then dilate the edges (fill all around). For example, if the patch size is 1024 and the step size is 512, then you need to fill in the four edges of the top, bottom, left and right with a step size of / 2, that is, 512 / 2 = 256 pixels. Use 1024×1024 as the sliding window and 512 as the step size. For each prediction, only the prediction result of the 512×512 center of the sliding window is retained (you can adjust the step size to a larger size, or retain a larger center area to improve efficiency). 3. Finally, perform the post-inference stitching, put these patches into the result map obtained after model inference, and stitch each of them in the center of the 512×512 area in order, crop the dilated part, and you can get the result map of the original size.
[0065] In one embodiment, S5 (obtaining the real-time position of the target based on the installation position of the second camera and the monocular vision positioning method) includes: based on the installation position of the second camera, correcting the posture of the second camera in the initial state, and obtaining the height of the second camera from the water surface; obtaining the azimuth and pitch angles of the pan-tilt rotation of the second camera when the target is photographed; based on the azimuth and pitch angles of the pan-tilt rotation, converting the coordinates of the target point to the initial state of the first camera or the second camera; calculating the azimuth and pitch angles of the target in the reference coordinate system; based on the height of the second camera from the water surface, and the azimuth and pitch angles of the target in the reference coordinate system, obtaining the coordinates of the target on the horizontal plane.
[0066] Figure 5 FIG. 1 is a schematic diagram of installing a second camera and finding a target provided by an exemplary embodiment of the present application. Figure 5 As shown in the figure, the second camera is installed parallel to the water surface (or the ground), that is, the optical axis of the camera is parallel to the horizontal plane, but there will be deviations during the actual installation process. Therefore, in order to achieve a good ranging effect, the posture in the initial state needs to be corrected. The pitch angle and roll angle of the second camera after installation can be measured by inertial navigation or other equipment, and then the posture of the second camera in the zero position state is corrected to be parallel to the water surface (or the ground). The conversion matrix is as shown in Formula 1:
[0067]
[0068] In formula 1, R r Indicates the rotation matrix required to correct the posture of the second camera in the zero position to be parallel to the water surface (or ground). represents the pitch angle rotation matrix, Represents the roll angle rotation matrix.
[0069] If the rotation order of the inertial navigation is ZXY, then:
[0070]
[0071] In formula 2 and formula 3, represents the pitch angle rotation matrix, Represents the roll angle rotation matrix, roll r Indicates the roll angle of the second camera, pitch r Indicates the pitch angle of the second camera.
[0072] Figure 6 is a schematic diagram of the principle of an imaging model of a second camera provided by an exemplary embodiment of the present application, such as Figure 6 As shown, the projection formula of the second camera can be derived through the imaging model as shown in Formula 4:
[0073]
[0074] In formula 4, u represents the horizontal axis in the image pixel coordinate system, v represents the vertical axis in the image pixel coordinate system, and f x Represents focal length, f y represents the focal length, (cx, cy) represents the center point of the second camera, X represents the horizontal axis in the camera coordinate system, Y represents the vertical axis in the camera coordinate system, K represents the camera intrinsic parameter matrix, Z (or ) represents the normalized parameter, and P represents the normalized coordinate.
[0075] From this we can deduce the normalized camera coordinates (Z = 1):
[0076]
[0077] In Formula 5, u represents the horizontal axis in the image pixel coordinate system, v represents the vertical axis in the image pixel coordinate system, and f x Represents focal length, f y represents the focal length, (cx, cy) represents the center point of the second camera, X represents the horizontal axis in the normalized coordinate system, and Y represents the vertical axis in the normalized coordinate system.
[0078] Therefore, if the pixel coordinates (u, v) of the target are detected from the image, the normalized coordinates P of the target under the changed coordinates can be calculated. C (XC , Y C , Z C ).
[0079] Since the second camera is constantly rotating up, down, left, and right, in order to achieve accurate positioning in this state, it is necessary to obtain the azimuth angle yaw of the gimbal rotation when acquiring the second image. c and pitch angle pirch c , and then transform the coordinates of the target point to the initial state of the second camera. No matter how the second camera rotates, the coordinates of the target in the reference coordinate system can be expressed as Formula 6:
[0080]
[0081] In formula 6, P w Represents a point in the world coordinate system, R r Indicates the rotation matrix required to correct the posture of the second camera in the zero position to be parallel to the water surface (or ground). represents the azimuth rotation matrix, represents the pitch angle rotation matrix, P c Represents a point in the normalized coordinate system.
[0082] From the rotation sequence YX of the second camera, we can know that:
[0083]
[0084]
[0085] In formula seven, Represents the azimuth rotation matrix, yaw c Indicates the azimuth angle of the pan / tilt rotation when acquiring the second image. In formula 8, The rotation matrix representing the pitch angle, pitch c Indicates obtaining the pitch angle of the gimbal rotation when acquiring the second image.
[0086] Get Pw(X w , Y w , Z w ), where P w Represents a point in the world coordinate system, X w Represents the horizontal axis in the world coordinate system, Y w Represents the vertical axis in world coordinates, Z w It represents the height in the world coordinate system. It should be noted that the coordinate scale of the target point is still unknown at this time, but the azimuth and pitch angle of the target in the reference coordinate system can be calculated:
[0087]
[0088] In formula 9, θ h represents the azimuth, θ v Indicates the pitch angle, X w Represents the horizontal axis in the world coordinate system, Y w Represents the vertical axis in world coordinates, Z w Indicates the height in the world coordinate system.
[0089] Since the height H of the second camera from the water surface (or ground) is known, assuming that the final target position uses the northeast sky as the reference coordinate system, the coordinates of the target on the horizontal plane are:
[0090]
[0091] In formula 10, θ h represents the azimuth, θ v Indicates the pitch angle, Y w It represents the vertical axis in the world coordinate system, X represents the horizontal axis in the northeast celestial coordinate system, Y represents the vertical axis in the northeast celestial coordinate system, and H represents the height of the second camera from the water surface (or the ground).
[0092] The target position is calculated in real time in the above manner and sent to the intelligent life-saving robot to guide the intelligent life-saving robot to approach the target. In addition, for target tracking, the PTZ (pitch, tail, zoom) of the second camera is fine-tuned by calculating the offset between the pixel coordinates of the target in the second image and the center point of the image in real time, thereby ensuring that the target is always in the center of the second image and achieving the tracking effect of the target.
[0093] Exemplary Devices
[0094] Figure 7 is a schematic diagram of the structure of a target positioning device provided by an exemplary embodiment of the present application. Figure 7 As shown, the target positioning device 6 includes: a first control module 61, which is used to control the first camera to shoot within a first preset range to obtain a first image; a second control module 62, which is used to control the second camera to shoot within a second preset range to obtain a second image; wherein the shooting area of the first camera is larger than the shooting area of the second camera; an identification module 63, which is used to process the first image and the second image through a dynamic sliding window virtual ring scanning technology to identify the target existence state; an adjustment module 64, which is used to adjust the second camera until the target exists in the second image when the target existence state indicates that the target exists in the first image and the target does not exist in the second image; a positioning module 65, which is used to obtain the real-time position of the target based on the installation position of the second camera and the monocular vision positioning method.
[0095] The target positioning device provided by the present application can effectively expand the target monitoring range and improve the target detection speed by linking the first camera with a large shooting range and the second camera. In addition, the dynamic sliding window virtual circular scanning technology can solve the problem that the target is difficult to identify due to the wide shooting range of the first camera, maintain the accuracy of target identification, reduce the possibility of target missed detection while not reducing the accuracy of target identification, reduce information processing time through monocular vision positioning, and further improve target positioning efficiency.
[0096] In one embodiment, the target positioning device 6 can also be configured as: when the target existence state indicates that the target exists in the second image and the target does not exist in the first image, based on the real-time position of the target, adjusting the second camera to track the target so as to place the target in the center of the second camera's shooting; or when the target existence state indicates that the target exists in the second image and the target exists in the first image, based on the real-time position of the target, adjusting the second camera to track the target so as to place the target in the center of the second camera's shooting.
[0097] In one embodiment, the first preset range may include a scene area within a first distance from the first camera, and the second preset range may include a scene area outside the first distance and within the second distance; or the first preset range may include the entire area within the scene, and the second preset range may include a fixed scene area.
[0098] In one embodiment, the recognition module 63 can be configured as follows: dividing the first image or the second image into multiple area images, and marking serial numbers on the multiple area images according to the order of segmentation; wherein the segmentation size of the area image is positively correlated with the distance of the target in the area image from the camera; placing the multiple area images into the object detection model for inference to obtain target position information of the multiple area images; based on the target position information and serial numbers of the multiple area images, restoring to the first image or the second image to obtain the target existence status.
[0099] In one embodiment, the recognition module 63 can be further configured as follows: adding an attention mechanism according to the probability of target appearance; based on the attention mechanism, performing frame skipping recognition on area images where the target has not appeared within a first preset period; based on the attention mechanism, performing frame-by-frame recognition on area images where the target has appeared a preset number of times within a second preset period.
[0100] In one embodiment, the recognition module 63 can be further configured to: after dividing the first image or the second image into multiple regional images, if the target is simultaneously located in at least two adjacent regional images, expand the first image or the second image containing the target so that the target is located in the same regional image.
[0101] In one embodiment, the recognition module 63 can be further configured to: fill the lower right border of the first image or the second image to an integer multiple of the sliding window prediction window size; based on a preset model recognition size, perform edge expansion on the filled first image or the second image; wherein, edge expansion means filling the four sides of the first image or the second image so that the size of the first image or the second image reaches the preset model recognition size, and the preset model recognition size represents the image size input to the object detection model for recognition.
[0102] In one embodiment, the positioning module 65 can be configured as follows: based on the installation position of the second camera, correct the posture of the second camera in the initial state, and obtain the height of the second camera from the water surface; obtain the azimuth and pitch angles of the pan-tilt rotation of the second camera when the target is photographed; based on the azimuth and pitch angles of the pan-tilt rotation, convert the coordinates of the target point to the initial state of the first camera or the second camera; calculate the azimuth and pitch angles of the target in the reference coordinate system; based on the height of the second camera from the water surface, and the azimuth and pitch angles of the target in the reference coordinate system, obtain the coordinates of the target on the horizontal plane.
[0103] Exemplary Electronic Devices
[0104] An electronic device includes: a processor; a memory for storing instructions executable by the processor; and the processor is used to execute the target positioning method described in the embodiment provided in the present application.
[0105] Below, reference Figure 8 The electronic device according to the embodiment of the present application is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signal from them.
[0106] Figure 8 A block diagram of an electronic device according to an embodiment of the present application is illustrated.
[0107] like Figure 8 As shown, the electronic device 100 includes one or more processors 110 and a memory 120 .
[0108] The processor 110 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.
[0109] The memory 120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 110 may run the program instructions to implement the target positioning method of each embodiment of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
[0110] In one example, the electronic device 100 may further include: an input device 130 and an output device 140 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0111] When the electronic device is a stand-alone device, the input device 130 may be a communication network connector for receiving collected input signals from the first device and the second device.
[0112] In addition, the input device 130 may also include, for example, a keyboard, a mouse, and the like.
[0113] The output device 140 can output various information to the outside, including the determined distance information, direction information, etc. The output device 140 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0114] Of course, to simplify, Figure 8 Only some of the components related to the present application in the electronic device 100 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 100 may also include any other appropriate components.
[0115] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0116] A computer-readable storage medium stores a computer program, and the computer program is used to execute the target positioning method described in the embodiment provided in this application.
[0117] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0118] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A target positioning method, characterized in that: include: Control the first camera to shoot within a first preset range to obtain a first image; Controlling the second camera to shoot within a second preset range to obtain a second image; wherein the shooting area of the first camera is larger than the shooting area of the second camera; The first image and the second image are processed by a dynamic sliding window virtual ring scanning technology to identify the existence status of the target; When the target existence state indicates that the target exists in the first image and the target does not exist in the second image, adjusting the second camera until the target exists in the second image; Based on the installation position of the second camera and the monocular vision positioning method, the real-time position of the target is obtained.
2. The target positioning method according to claim 1, characterized in that: Targeting methods also include: When the target existence state indicates that the target exists in the second image and does not exist in the first image, based on the real-time position of the target, adjusting the second camera to track the target so as to place the target at the center of the image captured by the second camera; or When the target existence state indicates that there is a target in the second image and there is a target in the first image, based on the real-time position of the target, the second camera is adjusted to track the target so as to place the target at the center of the second camera's shooting.
3. The target positioning method according to claim 1, characterized in that: The first preset range includes a scene area within a first distance from the first camera, and the second preset range includes a scene area outside the first distance and within a second distance; or The first preset range includes all areas within the scene, and the second preset range includes a fixed scene area.
4. The target positioning method according to claim 1, characterized in that: The first image and the second image are processed by the dynamic sliding window virtual ring scanning technology to identify the existence status of the target, including: The first image or the second image is divided into a plurality of region images, and the plurality of region images are marked with serial numbers according to the division order; wherein the division size of the region image is positively correlated with the distance of the target in the region image from the camera; Putting the plurality of area images into an object detection model for inference to obtain target position information of the plurality of area images; Based on the target position information and the sequence numbers of the plurality of area pictures, restore to the first picture or the second picture to obtain the target existence status.
5. The target positioning method according to claim 4, characterized in that: Putting the plurality of images of the region into the object detection model for inference, and obtaining target position information of the plurality of images of the region, including: Add attention mechanism according to the probability of target appearance; Based on the attention mechanism, skipping frames are performed on the pictures of the area where no target has appeared within a first preset period of time; Based on the attention mechanism, the area pictures in which the target has appeared a preset number of times within a second preset period are identified frame by frame.
6. The target positioning method according to claim 4, characterized in that: The first image or the second image is divided into a plurality of region images, and a serial number is marked on the plurality of region images according to the division order, including: After the first picture or the second picture is divided into a plurality of region pictures, if the target is simultaneously located in at least two adjacent region pictures, the first picture or the second picture containing the target is expanded so that the target is located in the same region picture.
7. The target positioning method according to claim 6, characterized in that: Dilation of the first picture or the second picture containing the target includes: Filling the lower right boundary of the first picture or the second picture to an integer multiple of the sliding window prediction window size; Based on a preset model recognition size, the first image or the second image after filling is edge-expanded; wherein the edge expansion means filling the four sides of the first image or the second image so that the size of the first image or the second image reaches the preset model recognition size, and the preset model recognition size represents the image size input to the object detection model for recognition.
8. The target positioning method according to claim 1, characterized in that: Based on the installation position of the second camera and the monocular vision positioning method, the real-time position of the target is obtained, including: Based on the installation position of the second camera, correct the posture of the second camera in the initial state, and obtain the height of the second camera from the water surface; Obtaining the azimuth and pitch angle of the pan / tilt rotation of the second camera when capturing the target; Based on the azimuth angle and the pitch angle of the pan / tilt rotation, convert the coordinates of the target point to an initial state of the first camera or the second camera; Calculate the azimuth and elevation angles of the target in the reference coordinate system; Based on the height of the second camera from the water surface, and the azimuth and pitch angle of the target in the reference coordinate system, the coordinates of the target on the horizontal plane are obtained.
9. A target positioning device, characterized in that: include: A first control module, used to control the first camera to shoot within a first preset range to obtain a first image; A second control module, used to control the second camera to shoot within a second preset range to obtain a second image; wherein the shooting area of the first camera is larger than the shooting area of the second camera; A recognition module, used to process the first image and the second image by using a dynamic sliding window virtual ring scanning technology to recognize the existence state of the target; an adjustment module, configured to adjust the second camera until the target exists in the second image when the target existence state indicates that the target exists in the first image and the target does not exist in the second image; The positioning module is used to obtain the real-time position of the target based on the installation position of the second camera and the monocular vision positioning method.
10. A target positioning system, characterized in that: include: A first camera, wherein the first camera is used to shoot within a first preset range to obtain a first image; A second camera, wherein the second camera is used to shoot within a second preset range to obtain a second image; Wherein, the shooting area of the first camera is larger than the shooting area of the second camera; A positioning device, the positioning device is communicatively connected with the first camera and the second camera, and is used to execute the target positioning method as described in any one of claims 1 to 8.
Citation Information
Cited By
Tea garden pest hazard identification method and equipment based on convolutional neural network
CN119888495A
