Camera linkage method and system based on joint calibration and trajectory prediction

Abstract

The invention discloses a camera linkage method and system based on joint calibration and trajectory prediction. The method comprises the following steps: performing event identification on a video stream collected by a first camera; extracting position coordinates of the target event in an image coordinate system of the first camera; according to the joint calibration relation, converting the position coordinates into PTZ parameters for controlling the second camera, and aligning the shooting center of the second camera with the target event; according to the position coordinate change of the target event in the video stream, calculating the motion track and the motion speed of the target event and predicting the future position coordinate of the target event; and inputting the future position coordinates into the joint calibration relation, generating an updated PTZ parameter, and controlling the second camera to track and shoot the target event. According to the invention, the problems of response lag and limited resolution caused by manual coordinate switching and lack of dynamic focal length adjustment in the prior art are solved, and automatic linkage and refined detail capture of the moving target under wide-area monitoring are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring and target tracking technology, and in particular to a camera linkage method and system based on joint calibration and trajectory prediction. Background Technology

[0002] In the field of intelligent monitoring and target tracking technology, multi-camera collaborative operation has become an important means to achieve wide-area coverage and detailed capture. However, existing technical solutions still have significant limitations: First, some systems lack dynamic tracking capabilities of focal length and cannot achieve real-time adaptive adjustment of zoom parameters, resulting in limited resolution of image recording of moving targets and difficulty in continuously acquiring fine details, especially when the target scale changes significantly, information loss is likely to occur; Second, other solutions rely on manually specifying spatial coordinate points to switch camera perspectives, without establishing an automatic triggering mechanism with event recognition algorithms, resulting in delayed linkage response and failing to meet the real-time processing needs of sudden events in complex scenarios.

[0003] The aforementioned problems have hindered the development of monitoring systems in the direction of intelligence and automation. There is an urgent need for a new linkage method that can integrate spatial calibration and trajectory prediction to achieve global perception of targets, intelligent focus adjustment, and event-driven real-time collaborative control.

[0004] Therefore, there is an urgent need to provide a technical solution to address the above problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a camera linkage method and system based on joint calibration and trajectory prediction.

[0006] In a first aspect, the present invention provides a camera linkage method based on joint calibration and trajectory prediction, the technical solution of which is as follows: Acquire the video stream captured by the first camera and perform event recognition on the video stream; When a target event is identified, the position coordinates of the target event in the image coordinate system of the first camera are extracted; Based on a pre-established joint calibration relationship, the position coordinates are converted into PTZ parameters for controlling the second camera; the joint calibration relationship includes the mapping relationship between the image coordinate system of the first camera and the physical space, and the set of PTZ parameters of the second camera corresponding to different physical space positions. The PTZ parameters are used to control the gimbal rotation and lens zoom of the second camera, so that the shooting center of the second camera is aligned with the target event; Based on the change in the position coordinates of the target event in the video stream, calculate the motion trajectory and speed of the target event, and predict the future position coordinates of the target event based on the motion trajectory and speed; The future location coordinates are input into the joint calibration relationship to generate updated PTZ parameters, and the second camera is controlled according to the updated PTZ parameters to track and capture the target event.

[0007] The beneficial effects of the camera linkage method based on joint calibration and trajectory prediction of the present invention are as follows: The method of the present invention automatically converts the image coordinates into the PTZ parameters of the second camera after the first camera identifies the target event by using the joint calibration relationship to control its alignment with the target, and combines trajectory prediction to update the PTZ parameters in real time to achieve zoom tracking. This solves the problems of response lag and resolution limitation caused by manual coordinate switching and lack of dynamic focal length adjustment in the existing methods, and realizes automatic linkage and fine detail capture of moving targets under wide-area monitoring.

[0008] Based on the above scheme, the camera linkage method based on joint calibration and trajectory prediction of the present invention can be further improved as follows.

[0009] In one alternative approach, the step of acquiring the video stream captured by the first camera and performing event recognition on the video stream includes: Receive the video stream continuously captured and transmitted by the first camera; The video stream is decoded to obtain consecutive image frames; The continuous image frames are analyzed frame by frame using a preset event recognition algorithm; When the event recognition algorithm identifies a target object that meets the preset conditions in the consecutive image frames, it is determined that the target event has been identified.

[0010] The advantages of adopting the above-mentioned optional method are: by continuously receiving video streams and decoding them into continuous image frames, and using preset algorithms to analyze target objects frame by frame, the accuracy and real-time performance of event recognition are improved, ensuring that emergencies can be captured in a timely manner.

[0011] In one alternative approach, the step of extracting the position coordinates of the target event in the image coordinate system of the first camera includes: Obtain the current image frame corresponding to the time when the target event is identified; Determine the image region occupied by the target object in the current image frame; Based on the boundary information of the image region, the position coordinates of the target event in the image coordinate system of the first camera are calculated.

[0012] The advantages of using the above optional method are: further calculating the position coordinates based on the occupied area and boundary information of the target object in the current image frame, realizing the accurate positioning of the target in the image coordinate system, and providing a reliable data foundation for subsequent coordinate transformation.

[0013] In one alternative approach, the step of converting the position coordinates into PTZ parameters for controlling the second camera based on a pre-established joint calibration relationship includes: The mapping relationship between the image coordinate system of the first camera and the physical space is determined; wherein the mapping relationship maps coordinate points in the image coordinate system to specific locations in the physical space; In the physical space, for multiple predetermined physical location points, multiple sets of PTZ parameters of the second camera are pre-calibrated to form the PTZ parameter set; wherein, each set of PTZ parameters in the PTZ parameter set is used to instruct the second camera to align the shooting center with the corresponding predetermined physical location point; Based on the location coordinates and the mapping relationship, determine the target physical location of the target event in the physical space; Based on the target physical location and the PTZ parameter set, the PTZ parameters used to control the second camera are determined.

[0014] The advantages of adopting the above optional method are: further pre-establishing the mapping relationship between the image coordinate system and the physical space and calibrating multiple sets of PTZ parameters, directly converting the target image coordinates into physical positions and corresponding PTZ parameters, and simplifying the control command generation process.

[0015] In one alternative approach, the step of controlling the gimbal rotation and lens zoom of the second camera using the PTZ parameters to align the shooting center of the second camera with the target event includes: The corresponding horizontal rotation angle, vertical rotation angle, and focal length values ​​are extracted from the PTZ parameters. Based on the horizontal and vertical rotation angles, the gimbal of the second camera is driven to rotate accordingly, and the focal length of the second camera lens is adjusted according to the focal length value so that the shooting center of the second camera is aligned with the physical location of the target event.

[0016] The advantages of using the above optional method are: further analyzing the horizontal rotation angle, vertical rotation angle and focal length value from the PTZ parameters, driving the gimbal to rotate and adjusting the lens focal length, so that the shooting center can be quickly aligned with the physical location of the target, and achieving precise visual locking.

[0017] In one alternative approach, the step of calculating the motion trajectory and velocity of the target event based on the change in the position coordinates of the target event in the video stream, and predicting the future position coordinates of the target event based on the motion trajectory and the motion velocity, includes: The position coordinates of the target event in multiple consecutive image frames of the video stream are obtained to form a position coordinate sequence; The motion speed of the target event is calculated based on the difference between adjacent position coordinates in the position coordinate sequence and the corresponding image frame time interval; Based on the position coordinate sequence, the motion trajectory of the target event in the video stream is generated from the time it is identified; By combining the motion trajectory and the motion speed, the future position coordinates of the target event at subsequent times are calculated.

[0018] The advantages of using the above-mentioned optional method are: further calculating the motion speed and trajectory based on the position coordinate sequence in continuous image frames, and predicting future position coordinates by combining time intervals, providing a quantitative basis for judging the target's motion trend and enhancing the tracking foresight.

[0019] In one alternative approach, the step of inputting the future position coordinates into the joint calibration relationship to generate updated PTZ parameters, and controlling the second camera to track and capture the target event based on the updated PTZ parameters, includes: Input the future location coordinates into the joint calibration relationship to determine the future physical location of the target event at a future time. Based on the future physical location and the PTZ parameter set, the updated PTZ parameters for controlling the second camera are determined; Based on the updated PTZ parameters, the gimbal and lens of the second camera are driven to make corresponding adjustments; During the continuous movement of the target event, the steps of predicting the future position coordinates, generating the updated PTZ parameters, and adjusting the drive of the second camera are repeatedly executed to complete the tracking and capturing of the target event.

[0020] The advantages of using the above optional method are: further inputting the predicted future position coordinates into the joint calibration relationship to generate updated PTZ parameters, repeating the prediction and adjustment during the continuous movement of the target, completing target tracking and shooting, and improving the degree of linkage automation.

[0021] Secondly, the present invention provides a camera linkage system based on joint calibration and trajectory prediction, the technical solution of which is as follows: The recognition module is used to acquire the video stream captured by the first camera and perform event recognition on the video stream; The extraction module is used to extract the position coordinates of the target event in the image coordinate system of the first camera when the target event is detected; The conversion module is used to convert the position coordinates into PTZ parameters for controlling the second camera according to a pre-established joint calibration relationship; the joint calibration relationship includes the mapping relationship between the image coordinate system of the first camera and the physical space, and the set of PTZ parameters of the second camera corresponding to different physical space positions. The control module is used to control the gimbal rotation and lens zoom of the second camera using the PTZ parameters, so as to align the shooting center of the second camera with the target event; The prediction module is used to calculate the motion trajectory and speed of the target event based on the change of the position coordinates of the target event in the video stream, and predict the future position coordinates of the target event based on the motion trajectory and the speed. The tracking module is used to input the future position coordinates into the joint calibration relationship, generate updated PTZ parameters, and control the second camera according to the updated PTZ parameters to track and capture the target event.

[0022] The beneficial effects of the camera linkage system based on joint calibration and trajectory prediction of the present invention are as follows: The system of the present invention automatically converts the image coordinates into the PTZ parameters of the second camera after the first camera identifies the target event, using the joint calibration relationship to control its alignment with the target. It also achieves zoom tracking by combining trajectory prediction to update the PTZ parameters in real time. This solves the problems of response lag and resolution limitation caused by manual coordinate switching and lack of dynamic focal length adjustment in existing systems, and realizes automatic linkage and fine detail capture of moving targets under wide-area monitoring.

[0023] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the camera linkage method based on joint calibration and trajectory prediction as described in this invention.

[0024] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the camera linkage method based on joint calibration and trajectory prediction of the present invention.

[0025] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0026] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of a camera linkage method based on joint calibration and trajectory prediction according to the present invention. Figure 2 This is a schematic diagram of the principle of a PTZ camera. Figure 3 This is a schematic diagram of the overall application architecture; Figure 4 This is a schematic diagram of an embodiment of a camera linkage system based on joint calibration and trajectory prediction according to the present invention. Figure 5 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0027] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0028] Figure 1 This diagram illustrates a flowchart of an embodiment of a camera linkage method based on joint calibration and trajectory prediction provided by the present invention. This camera linkage method based on joint calibration and trajectory prediction can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the camera linkage method based on joint calibration and trajectory prediction by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps: S1. Acquire the video stream captured by the first camera and perform event recognition on the video stream.

[0029] The first camera refers to a bullet camera with a fixed field of view, used for continuous monitoring and acquisition of video images over a large area; for example, a fixed surveillance camera installed on a highway gantry, facing a fixed lane for 24 / 7 recording. The video stream refers to a sequence of digital video data continuously acquired by cameras and transmitted in chronological order; for example, traffic road image data acquired and uploaded in real time by a bullet camera at a rate of 25 frames per second. Event recognition refers to the process of automatically analyzing video content through algorithms to detect predefined specific activities or states; for example, automatically identifying situations such as "vehicles illegally changing lanes" or "pedestrians trespassing" by analyzing bullet camera video streams using computer vision algorithms.

[0030] S2. When a target event is detected, extract the position coordinates of the target event in the image coordinate system of the first camera.

[0031] In this context, a target event refers to a specific event instance that is identified during the event recognition process and requires attention and tracking; for example, a white sedan illegally changing lanes in a video stream. An image coordinate system is a two-dimensional coordinate system established based on a single frame image captured by a camera, used to describe the position of pixels in the image; for example, a pixel coordinate system established with the top left corner of the image captured by a bullet camera as the origin (0,0), the positive X-axis pointing to the right, and the positive Y-axis pointing downwards. Position coordinates refer to the specific coordinate values ​​corresponding to the target event in the image coordinate system, used to quantify its position in the image; for example, the center point of the illegally changing lane vehicle is calculated to have coordinates of (450, 300) in the image coordinate system.

[0032] S3. Based on the pre-established joint calibration relationship, the position coordinates are converted into PTZ parameters for controlling the second camera; the joint calibration relationship includes the mapping relationship between the image coordinate system of the first camera and the physical space, and the set of PTZ parameters of the second camera corresponding to different physical space positions.

[0033] The joint calibration relationship refers to a pre-established set of data relationships used for mapping and transformation between the image coordinate system of the bullet camera, physical space, and the control parameters of the PTZ camera. For example, a parameter mapping relationship established through calibration experiments that maps the coordinates (450, 300) in the bullet camera image to a specific location on the actual road, and simultaneously maps them to the required rotation angle and zoom magnification of the PTZ camera. The second camera refers to a PTZ camera with pan-tilt-zoom (PTZ) capabilities, used for tracking and capturing details of a specific target. For example, a rotating surveillance camera (PTZ camera) mounted on the same gantry as the bullet camera, capable of horizontal and vertical rotation and possessing optical zoom. PTZ parameters refer to a set of parameters used to control the camera's pan-tilt-zoom (PTZ) rotation and lens zoom, typically including horizontal angle, vertical angle, and focal length. For example, a set of command parameters controlling the PTZ camera to rotate to a horizontal angle of 30°, a vertical angle of 15°, and adjust the lens focal length to 50mm. Physical space refers to the real three-dimensional world environment monitored by the camera; for example, a section of actual highway about 100 meters long and four lanes wide, jointly monitored by bullet cameras and PTZ cameras. PTZ parameter set refers to the collection of multiple sets of PTZ parameters pre-calibrated and stored for a series of predetermined locations in the physical space; for example, a calibration point is preset every 5 meters for the aforementioned highway section, and a set of PTZ parameters that allows the PTZ camera to be aimed at that point is stored. The set of all parameters constitutes the PTZ parameter set.

[0034] S4. Use the PTZ parameters to control the gimbal rotation and lens zoom of the second camera, and align the shooting center of the second camera with the target event.

[0035] Among these, pan-tilt rotation refers to the action of controlling the camera's pan-tilt head to rotate horizontally and vertically; for example, based on the PTZ parameters, driving the pan-tilt head of the PTZ camera to rotate 30° to the right and 15° downwards. Lens zoom refers to the action of adjusting the focal length of the camera lens to change the shooting field of view and image magnification; for example, based on the focal length value in the PTZ parameters, controlling the lens motor of the PTZ camera to drive the lens group to adjust the focal length from 10mm to 50mm, thereby zooming in on the image. Shooting center refers to the physical spatial orientation corresponding to the geometric center of the camera's image; for example, after the PTZ camera completes pan-tilt rotation and lens zoom, the point on the actual road that the center of its image is aligned with is the shooting center.

[0036] S5. Based on the change in the position coordinates of the target event in the video stream, calculate the motion trajectory and motion speed of the target event, and predict the future position coordinates of the target event based on the motion trajectory and motion speed.

[0037] The change in position coordinates refers to the difference in the position coordinates of a target event in the image coordinate system at different times. For example, the coordinates of a vehicle violating traffic rules are (450, 300) in the first second and (460, 295) in the next second; this difference reflects the change in position coordinates. The motion trajectory refers to the continuous sequence of position points formed by the target event in the image coordinate system or physical space over time. For example, the vehicle violating traffic rules moves from left to right in the image by connecting one coordinate point per second over 5 consecutive seconds. The motion speed refers to the rate and direction of position change of the target event per unit time, which can be calculated from the change in position coordinates and the time interval. For example, based on the change in the vehicle's coordinates between two consecutive frames and the time difference of 0.04 seconds between the two frames, the vehicle's movement speed in the image coordinate system is calculated to be 25 pixels per second. Future position coordinates refer to the prediction of the position of a target event in the image coordinate system at a certain point in the future, based on the current and historical motion state of the target event; for example, based on the current motion trajectory and speed of the vehicle, its coordinates in the image after 0.5 seconds are predicted to be approximately (485, 290).

[0038] S6. Input the future position coordinates into the joint calibration relationship to generate updated PTZ parameters, and control the second camera according to the updated PTZ parameters to track and capture the target event.

[0039] The updated PTZ parameters refer to the control parameters newly calculated based on the predicted future location of the target event, using a joint calibration relationship, to control the PTZ camera to track that future location. For example, based on the predicted future coordinates of the vehicle (485, 290), a new set of horizontal angle, vertical angle, and focal length values ​​are obtained by querying the joint calibration relationship. Tracking shooting refers to the process of continuously adjusting the PTZ parameters of the PTZ camera to ensure that its shooting center is always pointed at the moving target event in order to obtain a coherent and detailed image. For example, the PTZ camera continuously adjusts its angle and focal length according to the constantly updated PTZ parameters to ensure that the white vehicle violating the rules is always centered in the frame and clearly captured.

[0040] The technical solution of this embodiment automatically converts the image coordinates into the PTZ parameters of the second camera after the first camera recognizes the target event, using the joint calibration relationship to control its alignment with the target. It also combines trajectory prediction to update the PTZ parameters in real time to achieve zoom tracking. This solves the problems of response lag and resolution limitation caused by manual coordinate switching and lack of dynamic focal length adjustment in existing methods, and realizes automatic linkage and fine detail capture of moving targets under wide-area monitoring.

[0041] In one alternative approach, S1 specifically includes: Receive the video stream continuously captured and transmitted by the first camera.

[0042] The video stream is decoded to obtain consecutive image frames.

[0043] In this context, consecutive image frames refer to multiple static images that are closely adjacent in time sequence in a video stream; for example, in a video stream transmitted by a bullet camera, a series of single road images with timestamps T1, T2, T3...

[0044] The continuous image frames are analyzed frame by frame using a preset event recognition algorithm.

[0045] Event recognition algorithms refer to computer programs or models used to automatically analyze images or videos to detect specific events; for example, an algorithm based on a deep convolutional neural network, which is trained to recognize behaviors such as vehicle lane changes and parking in road scenes.

[0046] When the event recognition algorithm identifies a target object that meets the preset conditions in the consecutive image frames, it is determined that the target event has been identified.

[0047] The preset conditions refer to the rules or feature thresholds set in the event recognition algorithm to determine whether a target object constitutes a target event; for example, the algorithm sets "a vehicle crossing a solid white line for more than 0.5 seconds" as a preset condition for determining an "illegal lane change" event. The target object refers to the entity identified in the image frame that may constitute a target event; for example, a specific white sedan detected in the image frame is the concrete carrier for determining whether an "illegal lane change" event has occurred.

[0048] In the above-mentioned optional methods, the accuracy and real-time performance of event recognition are further improved by continuously receiving video streams and decoding them into continuous image frames, and then using a preset algorithm to analyze the target object frame by frame, ensuring that emergencies can be captured in a timely manner.

[0049] In one alternative approach, the step of extracting the position coordinates of the target event in the image coordinate system of the first camera includes: Obtain the current image frame corresponding to the time when the target event is identified.

[0050] The current image frame refers to the specific image frame corresponding to the first recognition or triggering of the target event during the event recognition process; for example, the road image being processed when the algorithm first determines that the white car meets the preset condition of "illegal lane change".

[0051] Determine the image region occupied by the target object in the current image frame.

[0052] The image region refers to the local area of ​​the image occupied by the target object in the current image frame, which is composed of pixels; for example, in the current image frame, the rectangular or irregularly shaped area composed of all the pixels surrounded by the outline of the white car.

[0053] Based on the boundary information of the image region, the position coordinates of the target event in the image coordinate system of the first camera are calculated.

[0054] Boundary information refers to data used to describe the extent of an image region, typically including the coordinates of the region's bounding rectangle or a set of contour points. For example, the boundary information describing the image region of a white car could be the coordinates of its top-left corner (440, 280) and bottom-right corner (460, 320) of its smallest bounding rectangle.

[0055] In the above optional methods, the position coordinates are further calculated based on the occupied area and boundary information of the target object in the current image frame, so as to achieve accurate positioning of the target in the image coordinate system and provide a reliable data foundation for subsequent coordinate transformation.

[0056] In one alternative approach, S3 specifically includes: The mapping relationship between the image coordinate system of the first camera and the physical space is determined; wherein the mapping relationship maps the coordinate points in the image coordinate system to specific locations in the physical space.

[0057] In the physical space, for multiple predetermined physical location points, multiple sets of PTZ parameters of the second camera are pre-calibrated to form the PTZ parameter set; wherein, each set of PTZ parameters in the PTZ parameter set is used to instruct the second camera to align the shooting center with the corresponding predetermined physical location point.

[0058] Among them, the predetermined physical location point refers to a specific location point in the physical space that is intentionally selected and used for system calibration; for example, in the joint calibration phase, a cross point is marked every 5 meters on the actual lane surface for calibration.

[0059] Based on the location coordinates and the mapping relationship, the target physical location corresponding to the target event in the physical space is determined.

[0060] The target physical location refers to the specific location of the target event in the actual physical space at the current moment. For example, calculated by joint calibration relationship, the illegal vehicle with coordinates (450, 300) in the image is located on the first lane about 50 meters directly below the gantry on the actual road.

[0061] Based on the target physical location and the PTZ parameter set, the PTZ parameters used to control the second camera are determined.

[0062] In the above optional methods, the mapping relationship between the image coordinate system and the physical space is further established in advance and multiple sets of PTZ parameters are calibrated. The target image coordinates are directly converted into physical positions and corresponding PTZ parameters, simplifying the control command generation process.

[0063] In one alternative approach, S4 specifically includes: The corresponding horizontal rotation angle, vertical rotation angle, and focal length values ​​are extracted from the PTZ parameters.

[0064] The horizontal rotation angle refers to the angle parameter controlling the pan-tilt head's horizontal rotation around the vertical axis; for example, the PTZ parameter indicates that the pan-tilt head needs to rotate 30° to the right from zero. The vertical rotation angle refers to the angle parameter controlling the pan-tilt head's pitch rotation around the horizontal axis; for example, the PTZ parameter indicates that the pan-tilt head needs to rotate 15° downwards from the horizontal position. The focal length value refers to the specific numerical parameter controlling the camera lens zoom motor to set the lens focal length; for example, the PTZ parameter indicates that the pan-tilt head needs to adjust the focal length from 10mm at the wide-angle end to 50mm.

[0065] Based on the horizontal and vertical rotation angles, the gimbal of the second camera is driven to rotate accordingly, and the focal length of the second camera lens is adjusted according to the focal length value so that the shooting center of the second camera is aligned with the physical location of the target event.

[0066] In the above-mentioned optional methods, the horizontal rotation angle, vertical rotation angle and focal length are further analyzed from the PTZ parameters to drive the gimbal to rotate and adjust the lens focal length, so that the shooting center can be quickly aligned with the physical location of the target and achieve precise visual locking.

[0067] In one alternative approach, S5 specifically includes: The position coordinates of the target event in multiple consecutive image frames of the video stream are obtained to form a position coordinate sequence.

[0068] Among them, the position coordinate sequence refers to the sequence formed by arranging the position coordinates of the target event in a continuous number of image frames in chronological order; for example, the coordinates of the illegal vehicle in the most recent 10 image frames are (450,300), (452,299), (455,298)..., and these coordinates constitute a position coordinate sequence.

[0069] The motion speed of the target event is calculated based on the difference between adjacent position coordinates in the position coordinate sequence and the corresponding image frame time interval.

[0070] Adjacent position coordinates refer to two coordinates that are temporally adjacent in a position coordinate sequence; for example, in the sequence, coordinates (452, 299) with timestamp T2 and coordinates (455, 298) with timestamp T3 are a pair of adjacent position coordinates. Image frame time interval refers to the time difference between the acquisition or processing of two consecutive frames in a video stream; for example, for a video stream with a frame rate of 25fps, the time interval between adjacent image frames is 0.04 seconds.

[0071] Based on the position coordinate sequence, the motion trajectory of the target event in the video stream is generated from the time it is identified.

[0072] By combining the motion trajectory and the motion speed, the future position coordinates of the target event at subsequent times are calculated.

[0073] In the above-mentioned optional methods, the motion speed and trajectory are further calculated based on the position coordinate sequence in consecutive image frames, and the future position coordinates are predicted in combination with the time interval, so as to provide a quantitative basis for judging the target motion trend and enhance the tracking foresight.

[0074] In one alternative approach, S6 specifically includes: The future location coordinates are input into the joint calibration relationship to determine the future physical location of the target event at a future time.

[0075] The future physical location refers to the predicted location of the target event in the actual physical space at a future time; for example, the approximate location of the violating vehicle on the actual road 0.5 seconds later, calculated based on the predicted future location coordinates and joint calibration relationship.

[0076] Based on the future physical location and the set of PTZ parameters, the updated PTZ parameters for controlling the second camera are determined.

[0077] Based on the updated PTZ parameters, the gimbal and lens of the second camera are driven to make corresponding adjustments.

[0078] During the continuous movement of the target event, the steps of predicting the future position coordinates, generating the updated PTZ parameters, and adjusting the drive of the second camera are repeatedly executed to complete the tracking and capturing of the target event.

[0079] In the above-mentioned optional methods, the predicted future position coordinates are further input into the joint calibration relationship to generate updated PTZ parameters. The prediction and adjustment are repeatedly performed during the continuous movement of the target to complete target tracking and shooting, thereby improving the degree of linkage automation.

[0080] Figure 2This describes the relative positional relationship between the PTZ camera and the target in physical space. The figure uses geometric figures to express the relationship between camera height, straight-line angle, lateral angle, straight-line distance from the road surface, and lateral distance from the road surface. Based on this geometric relationship, the distance calculation formula t² = (h * tan(a))² + n² is established, where h represents the camera height, a represents the camera straight-line angle, and n represents the lateral distance from the road surface. This relationship allows subsequent calculations to deduce the motion velocity based on the changes in the target event's position coordinates in the video stream, and finally smoothly generate updated PTZ parameters through a pre-established joint calibration relationship to control the PTZ camera to complete tracking and capturing.

[0081] Figure 3 This demonstrates the application architecture of this embodiment in a specific scenario. The architecture includes four fixed-field bullet cameras and one PTZ camera with gimbal turning and lens zoom capabilities. The workflow begins with the bullet cameras acquiring a video stream and performing event recognition. Once a target event is identified, its position coordinates in the corresponding bullet camera image coordinate system are extracted. These position coordinates are converted into PTZ parameters for controlling the PTZ camera based on a pre-established joint calibration relationship. The PTZ camera then performs gimbal turning and lens zoom according to the PTZ parameters, aligning the shooting center with the target event. Subsequently, based on the changes in the target event's position coordinates in consecutive image frames of the video stream, the motion trajectory and velocity are calculated, and the future position coordinates are predicted. The predicted future position coordinates are again input into the joint calibration relationship to generate updated PTZ parameters. The PTZ camera continuously adjusts according to the updated parameters, thereby achieving tracking and shooting of the target event and compensating for the blind spots of the bullet cameras.

[0082] Figure 4 This diagram illustrates a structural schematic of an embodiment of a camera linkage system 200 based on joint calibration and trajectory prediction provided by the present invention. Figure 4 As shown, the camera linkage system 200 based on joint calibration and trajectory prediction includes: The recognition module 201 is used to acquire the video stream captured by the first camera and perform event recognition on the video stream; Extraction module 202 is used to extract the position coordinates of the target event in the image coordinate system of the first camera when the target event is detected; The conversion module 203 is used to convert the position coordinates into PTZ parameters for controlling the second camera according to a pre-established joint calibration relationship; the joint calibration relationship includes the mapping relationship between the image coordinate system of the first camera and the physical space, and the set of PTZ parameters of the second camera corresponding to different physical space positions. The control module 204 is used to control the gimbal rotation and lens zoom of the second camera using the PTZ parameters, so as to align the shooting center of the second camera with the target event; The prediction module 205 is used to calculate the motion trajectory and motion speed of the target event based on the change of the position coordinates of the target event in the video stream, and predict the future position coordinates of the target event based on the motion trajectory and the motion speed; The tracking module 206 is used to input the future position coordinates into the joint calibration relationship, generate updated PTZ parameters, and control the second camera according to the updated PTZ parameters to track and capture the target event.

[0083] In one alternative embodiment, the identification module 201 is specifically used for: Receive the video stream continuously captured and transmitted by the first camera; The video stream is decoded to obtain consecutive image frames; The continuous image frames are analyzed frame by frame using a preset event recognition algorithm; When the event recognition algorithm identifies a target object that meets the preset conditions in the consecutive image frames, it is determined that the target event has been identified.

[0084] In an alternative embodiment, the extraction module 202 is specifically used for: Obtain the current image frame corresponding to the time when the target event is identified; Determine the image region occupied by the target object in the current image frame; Based on the boundary information of the image region, the position coordinates of the target event in the image coordinate system of the first camera are calculated.

[0085] In an alternative embodiment, the conversion module 203 is specifically used for: The mapping relationship between the image coordinate system of the first camera and the physical space is determined; wherein the mapping relationship maps coordinate points in the image coordinate system to specific locations in the physical space; In the physical space, for multiple predetermined physical location points, multiple sets of PTZ parameters of the second camera are pre-calibrated to form the PTZ parameter set; wherein, each set of PTZ parameters in the PTZ parameter set is used to instruct the second camera to align the shooting center with the corresponding predetermined physical location point; Based on the location coordinates and the mapping relationship, determine the target physical location of the target event in the physical space; Based on the target physical location and the PTZ parameter set, the PTZ parameters used to control the second camera are determined.

[0086] In an alternative embodiment, the control module 204 is specifically used for: The corresponding horizontal rotation angle, vertical rotation angle, and focal length values ​​are extracted from the PTZ parameters. Based on the horizontal and vertical rotation angles, the gimbal of the second camera is driven to rotate accordingly, and the focal length of the second camera lens is adjusted according to the focal length value so that the shooting center of the second camera is aligned with the physical location of the target event.

[0087] In an alternative embodiment, the prediction module 205 is specifically used for: The position coordinates of the target event in multiple consecutive image frames of the video stream are obtained to form a position coordinate sequence; The motion speed of the target event is calculated based on the difference between adjacent position coordinates in the position coordinate sequence and the corresponding image frame time interval; Based on the position coordinate sequence, the motion trajectory of the target event in the video stream is generated from the time it is identified; By combining the motion trajectory and the motion speed, the future position coordinates of the target event at subsequent times are calculated.

[0088] In an alternative embodiment, the tracking module 206 is specifically used for: Input the future location coordinates into the joint calibration relationship to determine the future physical location of the target event at a future time. Based on the future physical location and the PTZ parameter set, the updated PTZ parameters for controlling the second camera are determined; Based on the updated PTZ parameters, the gimbal and lens of the second camera are driven to make corresponding adjustments; During the continuous movement of the target event, the steps of predicting the future position coordinates, generating the updated PTZ parameters, and adjusting the drive of the second camera are repeatedly executed to complete the tracking and capturing of the target event.

[0089] It should be noted that the beneficial effects of the camera linkage system 200 based on joint calibration and trajectory prediction provided in the above embodiments are the same as those of the camera linkage method based on joint calibration and trajectory prediction described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0090] The camera linkage system 200 based on joint calibration and trajectory prediction of the present invention can be a computer program (including program code) running on a computer device. For example, the camera linkage system 200 based on joint calibration and trajectory prediction of the present invention is an application software that can be used to execute the corresponding steps in the camera linkage method based on joint calibration and trajectory prediction of the present invention.

[0091] In some embodiments, the camera linkage system 200 based on joint calibration and trajectory prediction of the present invention can be implemented in a combination of hardware and software. As an example, the camera linkage system 200 based on joint calibration and trajectory prediction of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the camera linkage method based on joint calibration and trajectory prediction of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0092] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0093] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned camera linkage methods based on joint calibration and trajectory prediction. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the camera linkage method based on joint calibration and trajectory prediction shown in any embodiment of the present invention by calling the computer program.

[0094] In one alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0095] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0096] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0097] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0098] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0099] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0100] It should be noted that, Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0101] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned camera linkage methods based on joint calibration and trajectory prediction.

[0102] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0103] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned camera linkage method based on joint calibration and trajectory prediction.

[0104] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0107] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0108] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0109] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0110] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0111] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A camera linkage method based on joint calibration and trajectory prediction, characterized in that, include: Acquire the video stream captured by the first camera and perform event recognition on the video stream; When a target event is identified, the position coordinates of the target event in the image coordinate system of the first camera are extracted; Based on a pre-established joint calibration relationship, the position coordinates are converted into PTZ parameters for controlling the second camera; the joint calibration relationship includes the mapping relationship between the image coordinate system of the first camera and the physical space, and the set of PTZ parameters of the second camera corresponding to different physical space positions. The PTZ parameters are used to control the gimbal rotation and lens zoom of the second camera, so that the shooting center of the second camera is aligned with the target event; Based on the change in the position coordinates of the target event in the video stream, calculate the motion trajectory and speed of the target event, and predict the future position coordinates of the target event based on the motion trajectory and speed; The future location coordinates are input into the joint calibration relationship to generate updated PTZ parameters, and the second camera is controlled according to the updated PTZ parameters to track and capture the target event.

2. The camera linkage method based on joint calibration and trajectory prediction according to claim 1, characterized in that, The step of acquiring the video stream captured by the first camera and performing event recognition on the video stream includes: Receive the video stream continuously captured and transmitted by the first camera; The video stream is decoded to obtain consecutive image frames; The continuous image frames are analyzed frame by frame using a preset event recognition algorithm; When the event recognition algorithm identifies a target object that meets the preset conditions in the consecutive image frames, it is determined that the target event has been identified.

3. The camera linkage method based on joint calibration and trajectory prediction according to claim 2, characterized in that, The step of extracting the position coordinates of the target event in the image coordinate system of the first camera includes: Obtain the current image frame corresponding to the time when the target event is identified; Determine the image region occupied by the target object in the current image frame; Based on the boundary information of the image region, the position coordinates of the target event in the image coordinate system of the first camera are calculated.

4. The camera linkage method based on joint calibration and trajectory prediction according to claim 3, characterized in that, The step of converting the position coordinates into PTZ parameters for controlling the second camera based on a pre-established joint calibration relationship includes: The mapping relationship between the image coordinate system of the first camera and the physical space is determined; wherein the mapping relationship maps coordinate points in the image coordinate system to specific locations in the physical space; In the physical space, for multiple predetermined physical location points, multiple sets of PTZ parameters of the second camera are pre-calibrated to form the PTZ parameter set; wherein, each set of PTZ parameters in the PTZ parameter set is used to instruct the second camera to align the shooting center with the corresponding predetermined physical location point; Based on the location coordinates and the mapping relationship, determine the target physical location of the target event in the physical space; Based on the target physical location and the PTZ parameter set, the PTZ parameters used to control the second camera are determined.

5. The camera linkage method based on joint calibration and trajectory prediction according to claim 1, characterized in that, The step of controlling the gimbal rotation and lens zoom of the second camera using the PTZ parameters to align the shooting center of the second camera with the target event includes: The corresponding horizontal rotation angle, vertical rotation angle, and focal length values ​​are extracted from the PTZ parameters. Based on the horizontal and vertical rotation angles, the gimbal of the second camera is driven to rotate accordingly, and the focal length of the second camera lens is adjusted according to the focal length value so that the shooting center of the second camera is aligned with the physical location of the target event.

6. The camera linkage method based on joint calibration and trajectory prediction according to any one of claims 1 to 5, characterized in that, The step of calculating the motion trajectory and velocity of the target event based on the change in the position coordinates of the target event in the video stream, and predicting the future position coordinates of the target event based on the motion trajectory and the motion velocity, includes: The position coordinates of the target event in multiple consecutive image frames of the video stream are obtained to form a position coordinate sequence; The motion speed of the target event is calculated based on the difference between adjacent position coordinates in the position coordinate sequence and the corresponding image frame time interval; Based on the position coordinate sequence, the motion trajectory of the target event in the video stream is generated from the time it is identified; By combining the motion trajectory and the motion speed, the future position coordinates of the target event at subsequent times are calculated.

7. The camera linkage method based on joint calibration and trajectory prediction according to claim 6, characterized in that, The step of inputting the future position coordinates into the joint calibration relationship to generate updated PTZ parameters, and controlling the second camera according to the updated PTZ parameters to track and capture the target event includes: Input the future location coordinates into the joint calibration relationship to determine the future physical location of the target event at a future time. Based on the future physical location and the PTZ parameter set, the updated PTZ parameters for controlling the second camera are determined; Based on the updated PTZ parameters, the gimbal and lens of the second camera are driven to make corresponding adjustments; During the continuous movement of the target event, the steps of predicting the future position coordinates, generating the updated PTZ parameters, and adjusting the drive of the second camera are repeatedly executed to complete the tracking and capturing of the target event.

8. A camera linkage system based on joint calibration and trajectory prediction, characterized in that, include: The recognition module is used to acquire the video stream captured by the first camera and perform event recognition on the video stream; The extraction module is used to extract the position coordinates of the target event in the image coordinate system of the first camera when the target event is detected; The conversion module is used to convert the position coordinates into PTZ parameters for controlling the second camera according to a pre-established joint calibration relationship; the joint calibration relationship includes the mapping relationship between the image coordinate system of the first camera and the physical space, and the set of PTZ parameters of the second camera corresponding to different physical space positions. The control module is used to control the gimbal rotation and lens zoom of the second camera using the PTZ parameters, so as to align the shooting center of the second camera with the target event; The prediction module is used to calculate the motion trajectory and speed of the target event based on the change of the position coordinates of the target event in the video stream, and predict the future position coordinates of the target event based on the motion trajectory and the speed. The tracking module is used to input the future position coordinates into the joint calibration relationship, generate updated PTZ parameters, and control the second camera according to the updated PTZ parameters to track and capture the target event.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the camera linkage method based on joint calibration and trajectory prediction as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, implements the camera linkage method based on joint calibration and trajectory prediction as described in any one of claims 1 to 7.

Images