Target positioning method and device, equipment and storage medium

Abstract

The invention discloses a target positioning method, device and equipment and a storage medium, and relates to the technical field of student standing tracking, and the method comprises the steps: carrying out the motion capture of a standing student through employing an AI figure detection algorithm, and determining the position information of the standing student in a panoramic camera; calculating a first offset from the standing student to the close-up camera according to pre-obtained calibration parameters and position information; coarse positioning of the standing student is realized according to the first offset, and a panoramic image and a close-up image after coarse positioning are acquired; and calculating a second offset from the standing student to the close-up camera according to the panoramic image and the close-up image, and controlling the close-up camera to a corresponding position based on the second offset to realize fine positioning. According to the method, the accuracy of target positioning of the close-up camera is ensured after coarse positioning and fine positioning, and the problem of limitation caused by fixed mounting angles among multiple cameras is effectively solved by combining an image processing technology and an intelligent algorithm.

Description

Technical Field

[0001] The present application relates to the field of tracking technology, and in particular to a target positioning method, device, equipment and storage medium. Background Art

[0002] In modern education scenarios, using panoramic wide-angle fixed-focus cameras combined with artificial intelligence (AI) technology to capture students standing up to answer questions in class has become an effective means to improve teaching interactivity and student participation. However, in this process, when it is necessary to use an optical zoom close-up camera to take more detailed and clearer photos of specific students, there are two major technical challenges: one is the lack of positioning accuracy, that is, how to accurately identify and lock the position of the target student from the panoramic image; the other is that the relative installation angles of multiple cameras are limited, which limits the system's ability to flexibly capture students in different positions. These problems not only affect the quality of the final video or picture, but also reduce the overall efficiency of the system and user experience.

[0003] Traditional solutions often rely on manual calibration or rule-based methods to improve positioning accuracy, but these solutions are usually difficult to adapt to complex classroom environment changes and are powerless when faced with multiple moving targets. In addition, due to the limitations of physical space layout, it is also very difficult to adjust the relative angles between multiple cameras to optimize coverage. Therefore, a new solution is urgently needed to overcome these limitations and ensure high-precision target positioning and high-quality image acquisition even in dynamically changing teaching environments. Summary of the invention

[0004] The main purpose of this application is to provide a target positioning method, device, equipment and storage medium, which aims to solve the technical problems of poor positioning accuracy of optical zoom close-up cameras in locating standing student targets and limited relative installation angles of multiple cameras in educational scenarios when using panoramic wide-angle fixed-focus cameras to capture standing student targets through AI technology.

[0005] To achieve the above object, the present application proposes a target positioning method, which is applied to a panoramic fixed-focus camera, wherein the panoramic fixed-focus camera includes a panoramic camera and a close-up camera, and the method includes: Using an AI person detection algorithm to capture the motion of the standing student, and determine the position information of the standing student in the panoramic camera; Calculating a first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information; Performing rough positioning of the standing student according to the first offset, and acquiring a panoramic image and a close-up image after rough positioning; A second offset from the standing student to the close-up camera is calculated according to the panoramic image and the close-up image after rough positioning, and the close-up camera is controlled to a corresponding position based on the second offset to complete the fine positioning of the standing student.

[0006] In one embodiment, before the step of calculating the first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information, the step further includes: Performing offline calibration on the panoramic camera and the close-up camera to obtain calibration parameters; The offline calibration of the panoramic camera and the close-up camera to obtain calibration parameters includes: Through the person detection algorithm, the panoramic camera and the close-up camera synchronously detect the standing students, and record the corresponding position information of the standing students into the calibration file.

[0007] In one embodiment, the step of calculating a first offset from the standing student to the close-up camera according to the calibration parameters and the position information comprises: According to the calibration parameters and position information, the mapping position of the standing pupil in the panoramic camera in the close-up camera is calculated by the homography transformation formula; Based on the mapped position, a first offset from the standing student to the close-up camera is calculated.

[0008] In one embodiment, the step of roughly locating the standing student according to the first offset and acquiring a panoramic image and a close-up image after the rough positioning includes: Converting the first offset into a first control instruction; Based on the first control instruction, the close-up camera is controlled to a corresponding position so that the standing student is located at the center of the screen of the close-up camera, thereby obtaining a panoramic image of the standing student and a close-up image after rough positioning.

[0009] In one embodiment, the step of calculating a second offset from the standing student to the close-up camera based on the panoramic image and the close-up image after the coarse positioning, and controlling the close-up camera to a corresponding position based on the second offset to complete the fine positioning of the standing student includes: Extracting feature points in the panoramic image and feature points in the close-up image after rough positioning; Performing feature point matching on the feature points in the panoramic image and the feature points in the roughly positioned close-up image to obtain matching point pairs; Based on the matching point pairs, calculating a homography matrix of the panoramic image and a homography matrix of the close-up image; Calculate a second offset from the standing pupil to the close-up camera according to the homography matrix of the panoramic image and the homography matrix of the close-up image; The step of calculating a second offset from the standing student to the close-up camera according to the homography matrix of the panoramic image and the homography matrix of the close-up image comprises: According to the homography matrix of the panoramic image and the homography matrix of the close-up image, the standing pupil in the panoramic camera is homographically transformed to obtain the coordinate position mapped on the close-up picture; Based on the coordinate position, calculate the second offset of the standing student to the close-up camera The close-up camera is controlled to a corresponding position based on the second offset to complete the detailed positioning of the students standing up.

[0010] In one embodiment, the step of controlling the close-up camera to a corresponding position based on the second offset to complete the detailed positioning of the standing student includes: Multi-task target detection is performed through a close-up camera to obtain the size of the standing students in the close-up picture; Based on the size, determining a preset zoom of the close-up camera; According to the second offset and the preset magnification, obtaining a second offset after the magnification is changed; The second offset after the magnification change is converted into a second control instruction, and based on the second control instruction, the close-up camera is controlled to a corresponding position to complete the detailed positioning of the standing student.

[0011] In one embodiment, after the step of controlling the close-up camera to a corresponding position based on the second offset to complete the detailed positioning of the standing student, the method further includes: Use the close-up camera to identify the standing student in the center of the close-up frame; Detecting the standing students by using target detection technology to obtain scale information of the standing students; Based on the scale information, the current close-up picture is composed to obtain a double-dashed line close-up picture.

[0012] In addition, to achieve the above purpose, the present application also proposes a target positioning device, the target positioning device comprising: A capture module is used to capture the motion of the standing student using an AI person detection algorithm and determine the position information of the standing student in the panoramic camera; A calculation module, the calculation module comprising a first calculation unit, a second calculation unit and a third calculation unit: The first calculation unit is used to calculate a first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information; The second calculation unit is used to roughly locate the standing student according to the first offset, and obtain a panoramic image and a close-up image after the rough positioning; The third calculation unit is used to calculate a second offset from the standing student to the close-up camera according to the panoramic image and the close-up image after coarse positioning, and control the close-up camera to a corresponding position based on the second offset to complete the fine positioning of the standing student.

[0013] In addition, to achieve the above objectives, the present application also proposes a target positioning device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target positioning method described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the target positioning method described above are implemented.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the target positioning method described above are implemented.

[0016] The present application implements a method, device, equipment and storage medium for target positioning, which is applied to a panoramic fixed-focus camera, which includes a panoramic camera and a close-up camera. The method includes: using an AI character detection algorithm to capture the motion of a standing student, and determining the position information of the standing student in the panoramic camera; calculating the first offset from the standing student to the close-up camera according to the calibration parameters and position information obtained in advance; performing rough positioning of the standing student according to the first offset, and obtaining a panoramic image and a close-up image after rough positioning; calculating the second offset from the standing student to the close-up camera according to the panoramic image and the close-up image after rough positioning, and controlling the close-up camera to the corresponding position based on the second offset, and completing the fine positioning of the standing student. After rough positioning and fine positioning, the method ensures the accuracy of the close-up camera's positioning target, and combines advanced image processing technology and intelligent algorithms to effectively overcome the limitation problem caused by the fixed installation angle between multiple cameras, allowing the system to flexibly adapt even in complex and changing environments, and realize high-precision target tracking and positioning functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the target positioning method of the present application; Figure 2 A schematic diagram of a calibration method for a panoramic camera and a close-up camera provided in Example 1 of the present application; Figure 3 A schematic diagram of the offset angle required for the first rough positioning of the close-up camera provided in the first embodiment of the present application; Figure 4 A schematic diagram of the target position of the close-up camera after rough positioning provided in Example 1 of the present application; Figure 5 A schematic diagram of feature point matching provided in Example 1 of the present application; Figure 6 A schematic diagram of a flow chart provided for the second embodiment of the target positioning method of the present application; Figure 7 A schematic diagram of the target position of the close-up camera after fine positioning provided in the second embodiment of the present application; Figure 8 A schematic diagram of a brief flow chart of a target positioning method provided in Example 1 of the present application; Fig. 9 This is a schematic diagram of the module structure of the target positioning device according to an embodiment of the present application; Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the target positioning method in the embodiment of the present application.

[0020] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: using an AI character detection algorithm to capture the motion of the standing student, and determine the position information of the standing student in the panoramic camera; calculating the first offset of the standing student to the close-up camera according to the pre-acquired calibration parameters and position information; performing coarse positioning of the standing student according to the first offset, and obtaining a panoramic image and a close-up image after coarse positioning; calculating the second offset of the standing student to the close-up camera according to the panoramic image and the close-up image after coarse positioning, and controlling the close-up camera to the corresponding position based on the second offset, to complete the fine positioning of the standing student.

[0024] The field of multi-camera positioning methods covers a variety of technologies and methods to ensure that image data obtained from different perspectives and sensors can be accurately aligned in space. With the development of technology, multi-camera positioning methods continue to evolve, from traditional image processing algorithms to innovative technologies based on deep learning, and researchers are pursuing higher positioning accuracy and speed.

[0025] Traditional methods usually rely on manually extracted feature matching, such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features), which have been widely used in remote sensing image processing. However, these methods that rely on manually designed features have limitations in processing complex and variable remote sensing data, especially when faced with significant geometric distortion and grayscale differences between different sensors. To this end, researchers have proposed structure-based registration methods, such as a multi-sensor remote sensing image registration method based on directional phase dense features (DFOP). This method captures the geometric structure distribution of the image by constructing a three-dimensional structural feature descriptor with illumination and contrast invariance, effectively resisting the grayscale differences between multi-sensor images and improving the matching accuracy.

[0026] In recent years, the rise of deep learning technology has brought new breakthroughs in multi-camera positioning methods. The powerful ability of deep neural networks in feature extraction enables deep learning-based registration methods to automatically learn more discriminative and robust feature expressions.

[0027] In modern education scenarios, using camera positions combined with artificial intelligence (AI) technology to capture students standing up to answer questions in class has become an effective means to improve teaching interactivity and student participation. However, in this process, when it is necessary to use an optical zoom close-up camera to take more detailed and clear shots of specific students, there are two major technical challenges: one is the lack of positioning accuracy, that is, how to accurately identify and lock the position of the target student from the panoramic image; the other is that the relative installation angle of multiple cameras is limited, which limits the system's ability to flexibly capture students in different positions. These problems not only affect the quality of the final video or picture, but also reduce the overall efficiency and user experience of the system. However, traditional solutions often rely on manual calibration or rule-based methods to improve positioning accuracy, but these solutions are usually difficult to adapt to complex changes in classroom environments and are powerless when facing multiple moving targets. In addition, due to the limitations of the physical space layout, it is also very difficult to adjust the relative angles between multiple cameras to optimize the coverage.

[0028] The present application provides a solution, which is applied to a panoramic fixed-focus camera, which includes a panoramic camera and a close-up camera. The solution includes: using an AI character detection algorithm to capture the motion of a standing student and determine the position information of the standing student in the panoramic camera; calculating the first offset from the standing student to the close-up camera based on the calibration parameters and position information obtained in advance; performing rough positioning of the standing student based on the first offset, obtaining a panoramic image and a close-up image after rough positioning; calculating the second offset from the standing student to the close-up camera based on the panoramic image and the close-up image after rough positioning, and controlling the close-up camera to the corresponding position based on the second offset to complete the fine positioning of the standing student. This method ensures the accuracy of the close-up camera's positioning target after rough positioning and fine positioning, and combines advanced image processing technology and intelligent algorithms to effectively overcome the limitation caused by the fixed installation angle between multiple cameras, allowing the system to flexibly adapt even in complex and changing environments, and achieve high-precision target tracking and positioning functions.

[0029] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, a camera device, etc., or an electronic device capable of realizing the above functions, etc. The following takes a personal computer as an example to illustrate this embodiment and the following embodiments.

[0030] Based on this, the present application embodiment provides a target positioning method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the target positioning method of the present application.

[0031] In this embodiment, the target positioning method includes steps S10 to S40: Step S10, using an AI person detection algorithm to capture the motion of the standing student, and determining the position information of the standing student in the panoramic camera; It should be noted that the method of this embodiment is applied to a panoramic fixed-focus camera, which includes a panoramic camera and a close-up camera. The panoramic camera is responsible for capturing a wide-angle image of the entire classroom or a specific area to ensure that the overall situation of all students can be captured; the close-up camera is used to capture detailed images of specific targets (such as students standing up). It has a smaller field of view, but can provide high-resolution local images.

[0032] In addition, it should be noted that AI task detection algorithm refers to the use of artificial intelligence technology, especially deep learning-based methods, to identify the location of people in images or videos. This algorithm can automatically separate the human body from the complex background and determine its precise location in the image.

[0033] Specifically, the panoramic camera captures images inside the classroom in real time and uses an AI person detection algorithm to analyze the images, identify students who stand up, and determine their location information in the panoramic image.

[0034] Step S20, calculating a first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information; It should be noted that the first offset is the motion adjustment information that the close-up camera needs to perform when performing the first phase of rough positioning in order to align with the position of the standing student.

[0035] In addition, it should be noted that offline calibration refers to a series of tests and adjustments performed on the camera in a controlled environment before actual use to determine the internal and external parameters of the camera.

[0036] It is understandable that since the lenses of panoramic cameras and close-up cameras may have inherent distortion and the specific parameters of each camera are different, step S201 is also included before performing step S20. Executing step S201 can avoid target positioning errors caused by lens distortion or inaccurate parameters, and improve the positioning accuracy and reliability of panoramic fixed-focus cameras in practical applications.

[0037] In a feasible implementation manner, step S20 may include step S201 before step S20: Step S201, performing offline calibration on the panoramic camera and the close-up camera to obtain calibration parameters; the performing offline calibration on the panoramic camera and the close-up camera to obtain calibration parameters includes: using a person detection algorithm to enable the panoramic camera and the close-up camera to synchronously detect standing students, and recording corresponding position information of the standing students into a calibration file.

[0038] Please refer to Figure 2 , Figure 2 The following is a schematic diagram of the calibration method for the panoramic camera and close-up camera provided in this embodiment. Figure 2 As shown, Figure 2 The panoramic camera is on the left, and the close-up camera is in the middle. During calibration, a standing student is used as the calibration object. When the calibration object walks in the public area, the panoramic camera and the close-up camera start recording or taking photos at the same time. At this time, the two cameras are synchronized in time to ensure that they can correspond to the data at the same time.

[0039] Use the AI ​​person detection algorithm to process the images obtained from the panoramic camera to determine the specific position of the person in the panoramic view. Also use the AI ​​person detection algorithm to process the images taken by the close-up camera to record the position information of the person in the close-up view. Compare the position information of the person from the two cameras. By comparing these position points, the relative posture relationship between the panoramic camera and the close-up camera can be calculated. And by selecting multiple sets of corresponding points, for example, the eight-point method requires at least 8 pairs of matching points, which are the positions of the same target that can be found in both the panoramic and close-up images. Use the eight-point method or other methods (such as RANSAC) to calculate the homography matrix H1 between the panoramic camera and the close-up camera. The homography matrix H1 describes how the target in the panoramic image is mapped to the projection relationship in the close-up image. Based on the above analysis results, create a calibration file containing all necessary parameters. The calibration file includes but is not limited to the homography matrix H1 and other necessary parameters.

[0040] Finally, additional testing is performed to verify the validity of the calibration results. If large deviations are found, it may be necessary to redo part or all of the calibration process until satisfactory accuracy is achieved.

[0041] Through the above steps, the panoramic camera and close-up camera are calibrated offline, which not only lays the foundation for subsequent precise positioning, but also uses the AI ​​person detection algorithm to make the entire calibration process more automated and efficient.

[0042] After the calibration parameters are obtained, the first offset from the standing student to the close-up camera can be calculated according to the calibration parameters and the position information of the standing student in the panoramic camera.

[0043] In another feasible implementation, step S20 may include steps S21-S22: Step S21, calculating the mapping position of the standing student in the panoramic camera in the close-up camera through a homography transformation formula according to the calibration parameters and the position information; Specifically, when the system starts, the pre-generated calibration file is read to obtain the homography matrix H1 and other necessary calibration parameters. Then, the position information of the standing student in the panoramic image is mapped to the position in the close-up image according to the homography matrix H1. The formula is as follows:

[0044] In the formula, and are homogeneous coordinates, , , represents the horizontal coordinate of the transformed standing student in the close-up picture, , , Represents the original horizontal coordinate of the standing student in the close-up picture before transformation. Represents the linear transformation parameters from one plane to another. These parameters together determine how to map points on one plane to another plane, where Determines the scaling and rotation of the transformed point in the x-axis direction, Determines the shearing and rotation of the transformed point in the x-axis direction, Determines the translation of the transformed point in the x-axis direction, Determines the shearing and rotation of the transformed point in the y-axis direction, Determines the scaling and rotation of the transformed point in the y-axis direction. Determines the translation of the transformed point in the y-axis direction, Determines the perspective deformation of the transformed point in the x-axis direction, Determines the perspective deformation of the transformed point in the y-axis direction, Usually set to 1 to maintain homogeneous coordinates.

[0045] Step S22: Based on the mapping position, a first offset from the standing student to the close-up camera is calculated.

[0046] Optionally, after obtaining the mapping position of the standing student in the panoramic camera in the close-up camera, the first offset of the first rough positioning of the close-up camera is calculated according to the following formula:

[0047] Where tanα and tanβ are the tangent function values ​​in the horizontal and vertical directions respectively. x_shift1 refers to the horizontal displacement distance of the standing pupil relative to the center of the close-up camera under the close-up camera perspective. img_width is the width of the close-up camera image.

[0048]

[0049] Where pan_angle1 is the parameter for adjusting the horizontal direction of the close-up camera in the first rough positioning. β is half of the horizontal field of view of the close-up camera, also known as the half field of view. pan_now1 is the current horizontal rotation angle of the close-up camera, that is, the direction the close-up camera is currently facing. img_width is the width of the close-up camera image. x_shift1 refers to the horizontal displacement distance of the standing pupil relative to the center of the close-up camera under the close-up camera's viewing angle.

[0050]

[0051] Where tanα and tanβ are the tangent function values ​​in the horizontal and vertical directions respectively. y_shift1 refers to the vertical displacement distance of the standing pupil relative to the center of the close-up camera under the current close-up camera viewing angle. img_height is the height of the close-up camera image.

[0052]

[0053] Where tit_angle1 is the parameter used to adjust the vertical direction of the close-up camera in the first rough positioning. β is half of the vertical field of view of the close-up camera, also known as the half field of view angle. tit_now1 is the current vertical rotation angle of the close-up camera, that is, the direction the close-up camera is currently facing. img_height1 is the height of the current close-up camera image.

[0054] Step S30, performing rough positioning of the standing student according to the first offset, and acquiring a panoramic image and a close-up image after rough positioning; Optionally, a first control instruction is generated according to the calculated pan_angle1 and tit_angle1. Then, the control instruction is sent to the pan / tilt head of the close-up camera through a control system, such as serial communication, network protocol, etc. Figure 3 As stated, Figure 3 The diagram is a schematic diagram of the offset angle required for the close-up camera in the first rough positioning. The pan / tilt adjusts the direction of the close-up camera according to the received first control instruction so that it is aimed at the standing student, completing the first rough positioning of the close-up camera. After the close-up camera is adjusted in place, the panoramic image of the standing student and the close-up image after rough positioning are collected from the panoramic camera and the close-up camera at the same time.

[0055] Through the above steps, the system can automatically calculate the angle that needs to be adjusted for the close-up camera and generate corresponding control instructions to adjust the camera direction, thereby completing the first rough positioning of the close-up camera.

[0056] Step S40, calculating a second offset from the standing student to the close-up camera according to the panoramic image and the close-up image after coarse positioning, and controlling the close-up camera to a corresponding position based on the second offset to complete fine positioning of the standing student.

[0057] like Figure 4 As shown, Figure 4 The following is a schematic diagram of the target position after the close-up camera is roughly positioned. It should be noted that the blue frame line is the panoramic camera field of view, and the green frame line is the close-up camera field of view. Due to the difference between the actual student height and the preset calibration personnel height, there may be a certain deviation in the first target rough positioning of the close-up camera, such as the deviation between the target person and the panoramic close-up center. If only one close-up camera rough positioning is performed, the close-up picture will magnify this error after zooming.

[0058] Therefore, this embodiment proposes a method to perform a second fine positioning of the close-up camera on the basis of the rough positioning of the close-up camera, thereby achieving the purpose of completing high-precision positioning in the case of unknown depth information.

[0059] In a feasible implementation, step S40 may include steps S41 to S45: Step S41, extracting feature points in the panoramic image and feature points in the roughly positioned close-up image; The close-up image and the panoramic image obtained after the first rough positioning in step S30 are used to extract feature points of the two images using an AI feature extraction algorithm.

[0060] Specifically, feature point detection algorithms are used, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (OrientedFAST and Rotated BRIEF), etc. These algorithms can identify key points in images and have certain invariance to scale, rotation and illumination changes. The selected feature extraction algorithm is applied to the panoramic image and the close-up image after coarse positioning to find the significant feature points in the two images.

[0061] For each detected feature point, a descriptor is calculated to characterize the local information around the point. The descriptor is usually a vector that reflects the gradient direction distribution or other patterns in the neighborhood of the feature point. The position of each feature point and its corresponding descriptor are recorded to prepare for the next step of feature matching.

[0062] Step S42, performing feature point matching on the feature points in the panoramic image and the feature points in the roughly positioned close-up image to obtain matching point pairs; Optionally, after obtaining the feature points of the panoramic image and the feature points of the close-up image, a feature matching algorithm, such as a brute force matching method, a FLANN matcher, etc., is used to find the most similar feature point pair between the two images. Figure 5 As shown, Figure 5 This is a schematic diagram of feature point matching. It should be noted that there are many pairs of matching feature points actually obtained. Figure 5 Only 4 pairs are drawn.

[0063] Step S43, calculating the homography matrix of the panoramic image and the homography matrix of the close-up image based on the matching point pairs; Optionally, based on the matching feature point pairs of the panoramic image and the close-up image, the homography matrix H2 from the panoramic image to the close-up image is estimated using the eight-point method. )and( ) to construct a linear equation system. Given more than 8 pairs of matching points, an overdetermined system of equations can be formed, which is then solved by SVD decomposition to obtain an estimate of the homography matrix H2.

[0064] Step S44, calculating a second offset from the standing student to the close-up camera according to the homography matrix of the panoramic image and the homography matrix of the close-up image; Optionally, the target position of the standing student in the panoramic camera is transformed by homography according to the homography matrix H2 obtained in step S43 to obtain the coordinate position mapped to the close-up picture. The homography transformation formula is:

[0065] In the formula, and are homogeneous coordinates, , , represents the horizontal coordinate of the transformed standing student in the close-up picture, , , Represents the original horizontal coordinate of the standing student in the close-up picture before transformation. Represents the linear transformation parameters from one plane to another. These parameters together determine how to map points on one plane to another plane, where Determines the scaling and rotation of the transformed point in the x-axis direction, Determines the shearing and rotation of the transformed point in the x-axis direction, Determines the translation of the transformed point in the x-axis direction, Determines the shearing and rotation of the transformed point in the y-axis direction, Determines the scaling and rotation of the transformed point in the y-axis direction. Determines the translation of the transformed point in the y-axis direction, Determines the perspective deformation of the transformed point in the x-axis direction, Determines the perspective deformation of the transformed point in the y-axis direction, Usually set to 1 to maintain homogeneous coordinates.

[0066] Then, the second offset is calculated according to the coordinate position. Similarly, the calculation formula is as follows:

[0067] Where tanα and tanβ are the tangent function values ​​in the horizontal and vertical directions, respectively. Refers to the horizontal displacement distance of the standing pupil relative to the center of the close-up camera under the close-up camera's perspective. img_width is the width of the close-up camera image.

[0068]

[0069] Where pan_angle2 is the parameter for adjusting the horizontal direction of the close-up camera in the second fine positioning. β is half of the horizontal field of view of the close-up camera, also known as the half field of view. pan_now is the current horizontal rotation angle of the close-up camera, that is, the direction the close-up camera is currently facing. img_width is the width of the close-up camera image. x_shift2 refers to the horizontal displacement distance of the standing pupil relative to the center of the close-up camera at the current close-up camera viewing angle.

[0070]

[0071] Where tanα and tanβ are the tangent function values ​​in the horizontal and vertical directions, respectively. Refers to the vertical displacement distance of the standing pupil relative to the center of the close-up camera under the close-up camera's perspective. img_height is the height of the close-up camera image.

[0072]

[0073] Where tit_angle2 is the parameter for adjusting the vertical direction of the close-up camera in the second fine positioning. β is half of the vertical field of view of the close-up camera, also known as the half field of view angle. tit_now2 is the current vertical rotation angle of the close-up camera, that is, the direction the close-up camera is currently facing. img_height2 is the height of the current close-up camera image. y_shift2 refers to the vertical displacement distance of the standing pupil relative to the center of the close-up camera under the current close-up camera viewing angle.

[0074] Through the above formula calculation, the second offset required for the close-up camera to perform a second fine positioning of the standing student is obtained.

[0075] Step S45, controlling the close-up camera to a corresponding position based on the second offset to complete the detailed positioning of the standing student.

[0076] Optionally, first, obtain the current close-up image from the current close-up camera, and use a multi-task target detection algorithm, such as YOLO, SSD or Faster R-CNN, to identify and frame the position of the standing student, and calculate the size of the standing student in the close-up image, including width and height.

[0077] Then, the size of the student standing up is compared with the preset target size. The preset target size can be set according to actual needs, for example, the standing student occupies a certain proportion of the close-up picture. Based on the ratio of the current standing student's size to the preset size, the zoom factor that needs to be adjusted is calculated. Assume that the current standing student's size is and , the default size is and , then the preset zoom_factor can be calculated according to the following formula: zoom_facto = =

[0078] In the formula, is the width of the student currently standing up, is the height of the student currently standing up, is the preset width of the standing students, The height for the preset standing students.

[0079] Since zooming will change the field of view of the close-up camera, it is necessary to adjust the second offset before zooming according to the preset zooming to obtain the second offset after zooming. Assume that the second offset after zooming is and , the focal length after zooming is focal_length_zoomed, then: = arctan

[0080] = arctan

[0081] In the formula, Indicates the parameters of the close-up camera after zoom adjustment in the horizontal direction during the second fine positioning process. Indicates the parameters of the close-up camera after zoom adjustment in the vertical direction during the second fine positioning process. It refers to the vertical displacement distance of the standing student relative to the center of the close-up camera screen under the current close-up camera viewing angle after magnification adjustment. It refers to the horizontal displacement distance of the standing student relative to the center of the close-up camera screen under the current close-up camera viewing angle after magnification adjustment.

[0082] based on and Generate second control instructions, which are used to instruct the close-up camera gimbal how to adjust the angle. Send the second control instructions to the close-up camera, and the close-up camera adjusts its horizontal and vertical angles according to the received second control instructions, and adjusts the zoom focal length at the same time, so that the standing student is not only located in the center of the close-up picture, but also meets the preset size.

[0083] Through the above-mentioned embodiment method, close-up positioning is divided into two steps. The first step is to perform coarse positioning according to the camera projection relationship. Since the target height and distance information are unknown under multiple camera positions, it is an ill-posed problem in projection positioning and cannot be accurately positioned. In the second step, after the close-up changes for the first time, the close-up picture and the panoramic picture are obtained, and the feature points are extracted and matched, and the homography matrix is ​​estimated. The panoramic target position is mapped to the close-up picture according to the homography matrix for secondary fine positioning; thereby achieving the purpose of completing high-precision positioning in the case of unknown depth information.

[0084] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 6 After step S40, the target positioning method further includes steps S401 to S403: Step S401, determining the standing student at the center of the close-up picture through the close-up camera; Step S402, using target detection technology to detect the standing student and obtain the scale information of the standing student; Step S403: compose the current close-up picture based on the scale information to obtain a double-dashed line close-up picture.

[0085] Please refer to Figure 7 , Figure 7 Figure 1 is a schematic diagram of the target position of the close-up camera after fine positioning. Figure 7 It can be seen that after the close-up camera has been finely positioned twice, the standing student has reached the exact center of the close-up picture. The close-up camera uses AI target detection technology to detect the target and obtain the scale information of the target at the center. Based on this information, the close-up camera composes the current picture and frames the close-up area with a green double dashed line. This not only realizes the function of the panoramic camera to capture the action, but also provides a high-definition close-up of the standing student through optical zoom.

[0086] For example, in order to help understand the implementation process of the target positioning method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 8 , Figure 8 A brief flowchart of a target positioning method is provided, specifically: In the panoramic fixed-focus camera, the panoramic camera and close-up camera are first calibrated to obtain the calibration parameters, and the calibration data is read to calculate the homography matrix H1. Then, the deep learning technology is used to capture the standing motion of the students in the classroom scene to obtain the position information of the standing students in the panoramic camera. Subsequently, based on the calibration parameters and position information, the homography matrix H1 is used to calculate the motion offset required from the close-up camera to the position of the standing students, that is, the first offset. According to the first offset, the close-up camera is controlled to the corresponding position to complete the rough positioning.

[0087] Since there is no depth information and the students are of different heights and in different positions relative to the camera, there will be a certain deviation in the first rough positioning of the target without precise positioning of the depth information. The second repositioning uses the deep learning method to detect and match the feature points of the close-up image and the panoramic image after the first close-up rough positioning, and calculates the homography matrix H2 based on the matched feature points. The offset of the second fine positioning, that is, the second offset, is calculated based on the homography matrix H2. Then, the zoom coefficient z is determined according to the size of the target in the close-up, and the second offset after the zoom is determined based on the zoom coefficient z. Finally, the close-up camera is controlled to the corresponding value according to the second offset after the zoom, and the fine positioning is completed. After the rough positioning and fine positioning, the accuracy of the close-up camera positioning the target is guaranteed.

[0088] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the target positioning method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0089] This application also provides a target positioning device, please refer to Fig. 9 , the target positioning device comprises: A capture module 10 is used to capture the motion of the standing student using an AI person detection algorithm, and determine the position information of the standing student in the panoramic camera; The calculation module 20 includes a first calculation unit, a second calculation unit and a third calculation unit: The first calculation unit is used to calculate a first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information; The second calculation unit is used to roughly locate the standing student according to the first offset, and obtain a panoramic image and a close-up image after the rough positioning; The third calculation unit is used to calculate a second offset from the standing student to the close-up camera according to the panoramic image and the close-up image after coarse positioning, and control the close-up camera to a corresponding position based on the second offset to complete the fine positioning of the standing student.

[0090] The target positioning device provided by the present application adopts the target positioning method in the above embodiment, which can solve the technical problems of poor positioning accuracy of the optical zoom close-up camera in locating standing student targets and limited relative installation angles of multiple cameras when using a panoramic wide-angle fixed-focus camera to capture standing student targets through AI technology in educational scenarios. Compared with the prior art, the beneficial effects of the target positioning device provided by the present application are the same as the beneficial effects of the target positioning method provided by the above embodiment, and the other technical features in the target positioning device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0091] The present application provides a target positioning device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target positioning method in the above-mentioned embodiment 1.

[0092] Reference below Fig.10 , which shows a schematic diagram of the structure of a target positioning device suitable for implementing the embodiment of the present application. The target positioning device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The target positioning device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0093] like Fig.10As shown, the target positioning device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the target positioning device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the target positioning device to communicate with other devices wirelessly or by wire to exchange data. Although the target positioning device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0094] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0095] The target positioning device provided by the present application adopts the target positioning method in the above embodiment, which can solve the technical problems of poor positioning accuracy of the optical zoom close-up camera in the process of locating standing student targets and limited relative installation angles of multiple cameras when using a panoramic wide-angle fixed-focus camera to capture standing student targets through AI technology in educational scenarios. Compared with the prior art, the beneficial effects of the target positioning device provided by the present application are the same as the beneficial effects of the target positioning method provided by the above embodiment, and the other technical features in the target positioning device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0096] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0098] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the target positioning method in the above-mentioned embodiment.

[0099] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0100] The computer-readable storage medium may be included in the target positioning device; or may exist independently without being assembled into the target positioning device.

[0101] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the target positioning device, the target positioning device: uses an AI character detection algorithm to capture the motion of the standing student and determines the position information of the standing student in the panoramic camera; calculates a first offset from the standing student to the close-up camera based on the calibration parameters and position information obtained in advance; performs coarse positioning of the standing student based on the first offset, and obtains a panoramic image and a close-up image after coarse positioning; calculates a second offset from the standing student to the close-up camera based on the panoramic image and the close-up image after coarse positioning, and controls the close-up camera to the corresponding position based on the second offset, thereby completing the fine positioning of the standing student.

[0102] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0105] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned target positioning method, and can solve the technical problems of poor positioning accuracy of optical zoom close-up cameras in locating standing student targets and limited relative installation angles of multiple cameras when using panoramic wide-angle fixed-focus cameras to capture standing student targets through AI technology in educational scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the target positioning method provided by the above-mentioned embodiments, and will not be elaborated here.

[0106] The present application also provides a computer program product, including a computer program, which implements the steps of the target positioning method as described above when executed by a processor.

[0107] The computer program product provided by the present application can solve the technical problems of poor positioning accuracy of optical zoom close-up cameras in locating standing student targets and limited relative installation angles of multiple cameras when using a panoramic wide-angle fixed-focus camera to capture standing student targets through AI technology in educational scenarios. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the target positioning method provided in the above embodiments, and will not be elaborated here.

[0108] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A target positioning method, characterized in that: The method is applied to a panoramic fixed-focus camera, which includes a panoramic camera and a close-up camera. The method includes: Using an AI person detection algorithm to capture the motion of the standing student, and determine the position information of the standing student in the panoramic camera; Calculating a first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information; Perform a rough positioning of the standing student according to the first offset, and obtain a panoramic image and a close-up image after the rough positioning; A second offset from the standing student to the close-up camera is calculated according to the panoramic image and the close-up image after rough positioning, and the close-up camera is controlled to a corresponding position based on the second offset to complete the fine positioning of the standing student.

2. The method according to claim 1, characterized in that Before the step of calculating the first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information, the method further includes: Performing offline calibration on the panoramic camera and the close-up camera to obtain calibration parameters; The offline calibration of the panoramic camera and the close-up camera to obtain calibration parameters includes: Through the person detection algorithm, the panoramic camera and the close-up camera synchronously detect the standing students, and record the corresponding position information of the standing students into the calibration file.

3. The method according to claim 1, characterized in that The step of calculating a first offset from the standing student to the close-up camera according to the calibration parameters and the position information comprises: According to the calibration parameters and position information, the mapping position of the standing pupil in the panoramic camera in the close-up camera is calculated by the homography transformation formula; Based on the mapped position, a first offset from the standing student to the close-up camera is calculated.

4. The method according to claim 3, characterized in that The step of roughly locating the standing student according to the first offset and acquiring a panoramic image and a close-up image after the rough positioning comprises: Converting the first offset into a first control instruction; Based on the first control instruction, the close-up camera is controlled to a corresponding position so that the standing student is located at the center of the screen of the close-up camera, thereby obtaining a panoramic image of the standing student and a close-up image after rough positioning.

5. The method according to claim 1, characterized in that The steps of calculating a second offset from the standing student to the close-up camera according to the panoramic image and the close-up image after the rough positioning, and controlling the close-up camera to a corresponding position based on the second offset, and completing the fine positioning of the standing student include: Extracting feature points in the panoramic image and feature points in the close-up image after rough positioning; Performing feature point matching on the feature points in the panoramic image and the feature points in the roughly positioned close-up image to obtain matching point pairs; Based on the matching point pairs, calculating a homography matrix of the panoramic image and a homography matrix of the close-up image; Calculate a second offset from the standing pupil to the close-up camera according to the homography matrix of the panoramic image and the homography matrix of the close-up image; The step of calculating a second offset from the standing pupil to the close-up camera according to the homography matrix of the panoramic image and the homography matrix of the close-up image comprises: According to the homography matrix of the panoramic image and the homography matrix of the close-up image, the standing pupil in the panoramic camera is homographically transformed to obtain the coordinate position mapped on the close-up picture; Based on the coordinate position, calculating a second offset from the standing student to the close-up camera; The close-up camera is controlled to a corresponding position based on the second offset to complete the detailed positioning of the students standing up.

6. The method according to claim 5, characterized in that The step of controlling the close-up camera to a corresponding position based on the second offset to complete the detailed positioning of the standing student comprises: Multi-task target detection is performed through a close-up camera to obtain the size of the standing students in the close-up picture; Based on the size, determining a preset zoom of the close-up camera; According to the second offset and the preset magnification, obtaining a second offset after the magnification is changed; The second offset after the magnification change is converted into a second control instruction, and based on the second control instruction, the close-up camera is controlled to a corresponding position to complete the detailed positioning of the standing student.

7. The method according to claim 1, characterized in that After the step of controlling the close-up camera to a corresponding position based on the second offset to complete the detailed positioning of the standing students, the method further includes: Use the close-up camera to identify the standing student in the center of the close-up frame; Detecting the standing students by using target detection technology to obtain scale information of the standing students; Based on the scale information, the current close-up picture is composed to obtain a double-dashed line close-up picture.

8. A target positioning device, characterized in that: The device comprises: A capture module is used to capture the motion of the standing student using an AI person detection algorithm and determine the position information of the standing student in the panoramic camera; A calculation module, the calculation module comprising a first calculation unit, a second calculation unit and a third calculation unit: The first calculation unit is used to calculate a first offset from the standing student to the close-up camera according to the pre-acquired calibration parameters and the position information; The second calculation unit is used to roughly locate the standing student according to the first offset, and obtain a panoramic image and a close-up image after the rough positioning; The third calculation unit is used to calculate a second offset from the standing student to the close-up camera according to the panoramic image and the close-up image after coarse positioning, and control the close-up camera to a corresponding position based on the second offset to complete the fine positioning of the standing student.

9. A target positioning device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target positioning method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the target positioning method according to any one of claims 1 to 7 are implemented.

Images