Multi-view object detection and model training methods, systems, devices and media

Abstract

The present invention provides a multi-view target detection and model training method, system, device, and medium. The detection method includes: synchronously capturing multiple video images to obtain intrinsic and extrinsic parameter matrices of corresponding cameras; inputting the multiple video images, the intrinsic and extrinsic parameter matrices into a disparity estimation network of the multi-view target detection model, extracting image features, detecting targets in each video image, fusing the image features of each channel, the intrinsic and extrinsic parameter matrices, and generating a disparity map; inputting the disparity map, the multiple video images, the intrinsic and extrinsic parameter matrices into a target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video images using the intrinsic and extrinsic parameter matrices, obtaining two-dimensional annotation results for each target from the corrected video images, and generating three-dimensional coordinates for each target based on the corrected video images and the disparity map. This method effectively integrates data from different viewpoints, making target detection and recognition more accurate and reliable.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a multi-view target detection and model training method, system, equipment and medium. Background Art

[0002] Currently, multi-view object detection methods are gradually gaining attention. These methods fuse views from multiple cameras to provide richer spatial information, thereby significantly improving the accuracy of object detection. Such multi-view and multi-sensor fusion methods improve the accuracy of object detection, but they often require a lot of computing resources when processing large-scale data and complex models. This is particularly prominent in resource-constrained scenarios, such as mobile devices, real-time systems, or large-scale camera networks, where computing resources and storage capabilities are limited. In addition, existing technologies often lack mechanisms to effectively coordinate and fuse information between multiple cameras, especially when the number of cameras increases, which has a more significant impact on system performance and resource requirements. Therefore, it is necessary to provide a method, system, device, and medium for multi-view object detection and model training. Summary of the Invention

[0003] The present invention provides a multi-view target detection method to solve the problem in the prior art that when multiple cameras are coordinated, there is a lack of an effective association mechanism, resulting in the inability to effectively utilize data from different viewpoints.

[0004] The present invention provides a multi-view target detection method, comprising: synchronously capturing multiple video images and obtaining intrinsic parameter matrices and extrinsic parameter matrices of corresponding cameras; inputting the multiple video images and the corresponding intrinsic parameter matrices and extrinsic parameter matrices into a disparity estimation network of a multi-view target detection model, extracting image features of each video image, detecting targets in each video image, and fusing the extracted image features and the corresponding intrinsic parameter matrices and extrinsic parameter matrices to generate a disparity map; inputting the disparity map, the multiple video images and the corresponding intrinsic parameter matrices and extrinsic parameter matrices into a target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video images using the intrinsic parameter matrices and extrinsic parameter matrices, obtaining a two-dimensional annotation result of each target from the corrected video images, and generating three-dimensional coordinates of each target based on the corrected video images and the disparity map.

[0005] In one embodiment of the present invention, the method of inputting multiple video images and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the disparity estimation network of the multi-view target detection model, extracting image features of each video image, detecting targets in each video image, and fusing the extracted image features and corresponding intrinsic parameter matrices and extrinsic parameter matrices to generate a disparity map includes: inputting multiple video images and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the disparity estimation network, extracting image features of each video image, and obtaining image features of each video image; detecting targets in each video image based on the extracted image features of each video image; for each target: performing feature matching on the extracted image features of each image, and obtaining image features of the same target under the perspective of each camera; and calculating the disparity value of the same target based on the image features of the same target under the perspective of each camera, and the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera; and generating a disparity map based on the calculated disparity value of each target.

[0006] In one embodiment of the present invention, the disparity value of the same target is calculated based on the image features of the same target under different camera perspectives, and the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera, including: identifying and recording the image coordinates of each image feature of the same target under the corresponding camera perspective; converting the image coordinates of each image feature into three-dimensional coordinates in the corresponding camera based on the intrinsic parameter matrix of the camera; converting the three-dimensional coordinates of each image feature into world coordinates in a world coordinate system based on the extrinsic parameter matrix of the camera; calculating the difference between the world coordinates corresponding to each image feature, and using the calculated difference as the disparity value of the same target.

[0007] In one embodiment of the present invention, the disparity map, multiple video images and corresponding intrinsic parameter matrix and extrinsic parameter matrix are input into the target tracking network of the multi-view target detection model, the intrinsic parameter matrix and the extrinsic parameter matrix are used to perform spatial correction of the corresponding video images, the two-dimensional annotation result of each target is obtained from the corrected video images, and the three-dimensional coordinates of each target are generated based on the corrected video images and the disparity map, including: inputting the disparity map, multiple video images and corresponding intrinsic parameter matrix and extrinsic parameter matrix into the target tracking network, correcting the distortion of each video image based on the intrinsic parameter matrix and the extrinsic parameter matrix, and aligning each video image to a unified preset reference perspective; generating a corresponding two-dimensional annotation result for each target in the corrected video image; wherein the two-dimensional annotation result includes the attributes and size information of the corresponding target; and generating the three-dimensional coordinates of each target based on the two-dimensional annotation result and the disparity map of each target.

[0008] In one embodiment of the present invention, the process of spatial correction of the video picture includes: based on the image distortion correction algorithm, adjusting the position of each pixel point in the corresponding video picture according to the distortion coefficient in the intrinsic parameter matrix; performing perspective transformation on the adjusted video picture based on the corresponding intrinsic parameter matrix; and using the corresponding extrinsic parameter matrix to perform position correction on the video picture after perspective transformation.

[0009] In one embodiment of the present invention, after obtaining the two-dimensional annotation results of each target in each video screen based on the corrected video screen and the disparity map, and generating the three-dimensional coordinates of each target, it also includes: displaying the two-dimensional annotation results of each target in the corresponding video screen.

[0010] In one embodiment of the present invention, a training method for a multi-view target detection model is also provided, the method comprising: obtaining a synchronously acquired multi-channel video frame set, and a two-dimensional annotation result set and a three-dimensional coordinate set corresponding to each channel of the video frame set; inputting the multi-channel video frame set, the corresponding intrinsic parameter matrix and the extrinsic parameter matrix into a disparity estimation network of the multi-view target detection model, extracting an image feature set of each channel of the video frame set, detecting the target in each channel of the video frame, and fusing the extracted image feature set, the intrinsic parameter matrix and the extrinsic parameter matrix to generate a disparity atlas; and fusing the disparity atlas, the multi-channel video frame set, the intrinsic parameter matrix and the extrinsic parameter matrix to generate a disparity atlas. The parameter matrix and the external parameter matrix are input into the target tracking network of the multi-view target detection model, and the spatial correction of the corresponding video picture set is performed using the internal parameter matrix and the external parameter matrix. The two-dimensional annotation result of each target is obtained from the corrected video picture, and the predicted three-dimensional coordinate set of each target is generated based on the corrected video picture set and the disparity atlas; based on the difference between the predicted two-dimensional annotation result set and the two-dimensional annotation result set, and the difference between the predicted three-dimensional coordinate set and the three-dimensional coordinate set, the parameters of the multi-view target detection model are updated to obtain a trained multi-view target detection model.

[0011] In one embodiment of the present invention, a system for multi-view target detection is also provided, which includes: a data acquisition module for synchronously capturing multiple video images and obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera; a disparity map generation module for inputting the multiple video images and the corresponding intrinsic parameter matrix and extrinsic parameter matrix into the disparity estimation network of the multi-view target detection model, extracting the image features of each video image, detecting the target in each video image, and fusing the extracted image features of each image and the corresponding intrinsic parameter matrix and extrinsic parameter matrix to generate a disparity map; a target detection module for inputting the disparity map, multiple video images and the corresponding intrinsic parameter matrix and extrinsic parameter matrix into the target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video image using the intrinsic parameter matrix and extrinsic parameter matrix, obtaining the two-dimensional annotation result of each target from the corrected video image, and generating the three-dimensional coordinates of each target based on the corrected video image and the disparity map.

[0012] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the steps of any of the multi-view target detection methods or multi-view target detection model training methods described above.

[0013] In one embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a computer processor, the computer executes the steps of any of the multi-view target detection methods or multi-view target detection model training methods described above.

[0014] The present invention proposes a multi-perspective target detection method. By synchronously collecting multiple video images and combining the intrinsic and extrinsic parameter matrices of each camera, the system can obtain more comprehensive spatial information, thereby effectively integrating data from different perspectives, making target detection and recognition more accurate and reliable. By using the disparity estimation network to extract image features and generate a disparity map, the position difference of the target under different camera perspectives can be accurately calculated, which helps to accurately locate the position of the target in three-dimensional space. The disparity map and multi-channel video data are input into the target tracking network and spatially corrected, which can effectively track the changes of the target in different time and space. In addition, the present invention also generates the three-dimensional coordinates of the target for subsequent analysis. Through this multi-perspective target detection and tracking method, more comprehensive and accurate spatial analysis and target processing can be achieved, providing strong technical support for various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a flow chart of a multi-view target detection method provided by an embodiment of the present invention;

[0016] Figure 2 Shown is a schematic diagram of the test results in one embodiment of the present invention;

[0017] Figure 3 Shown is a model architecture diagram of a multi-view object detection model according to an embodiment of the present invention;

[0018] Figure 4 Shown is a structural block diagram of a system for multi-view target detection provided by an embodiment of the present invention;

[0019] Figure 5 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0021] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0023] The present invention provides a multi-perspective target detection method. By synchronously capturing multiple video images and combining the intrinsic and extrinsic parameter matrices of each camera, the system can obtain more comprehensive spatial information, thereby effectively integrating data from different perspectives, making target detection and recognition more accurate and reliable. By using a disparity estimation network to extract image features and generate a disparity map, the position difference of the target under different camera perspectives can be accurately calculated, which helps to accurately locate the position of the target in three-dimensional space. The disparity map and multi-channel video data are input into a target tracking network and spatially corrected, which can effectively track the changes of the target in different time and space. In addition, the present invention also generates the three-dimensional coordinates of the target for subsequent analysis. Through this multi-perspective target detection and tracking method, more comprehensive and accurate spatial analysis and target processing can be achieved, providing strong technical support for various application scenarios. The algorithm of the present invention combines the existing two-dimensional target detection model architecture with binocular camera measurement theory as the basis. It performs spatial association of multiple cameras through the target space relationship matrix and applies it to the loss function, thereby ensuring that the model has the ability to output spatial three-dimensional information while outputting single-frame image detection targets separately.

[0024] See Figure 1 ,Multi-view target detection includes the following steps:

[0025] S1. Synchronously capture multiple video images and obtain the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera.

[0026] The experimental videos recorded by cameras at different positions are synchronously acquired, and continuous video images are extracted from them. At the same time, the intrinsic parameter matrix and extrinsic parameter matrix of each camera need to be extracted. Among them, the intrinsic parameter matrix of the camera includes parameters such as the focal length, optical center position and distortion coefficient of the camera, which are used to characterize the attribute information of the camera itself. The extrinsic parameter matrix of the camera includes the position (i.e., translation vector) and orientation (i.e., rotation matrix) of the camera relative to the common world coordinate system. The characteristics and imaging properties of the camera can be understood through the intrinsic parameter matrix, and the spatial position relationship between the cameras can be determined through the extrinsic parameter matrix, so as to facilitate more accurate analysis of the video image information later. It should be noted that the intrinsic parameter matrix and extrinsic parameter matrix of the camera can be obtained in advance based on a prior calibration process.

[0027] S2. Input multiple video images and the corresponding intrinsic parameter matrix and extrinsic parameter matrix into the disparity estimation network of the multi-view target detection model, extract the image features of each video image, detect the target in each video image, and fuse the extracted image features and the corresponding intrinsic parameter matrix and extrinsic parameter matrix to generate a disparity map.

[0028] It should be noted that the multi-view target detection model in this application is applicable to various models capable of target detection, including but not limited to Yolov5, Yolov6, Yolov7, etc., without specific limitation. Multi-channel video images and corresponding internal parameter matrices and external parameter matrices are input into the disparity estimation network, and key image features such as image edges, corners and textures are extracted from each video image, so as to achieve target detection more accurately in the future. The target in the corresponding video image is detected using the extracted image features, and the disparity map is calculated by fusing the image feature data from all video sources and combining the corresponding internal parameter matrix and external parameter matrix. Among them, the disparity map is generated according to the relative position change of the same target in different video sources, which reflects the depth change of the same scene or object in space observed from different camera angles. This change is determined by the spatial information of the internal and external parameter matrices and the image features of the multi-channel video images.

[0029] In one embodiment of the present invention, the method of inputting multiple video images and corresponding intrinsic parameter matrices and extrinsic parameter matrices into a disparity estimation network of a multi-view target detection model, extracting image features of each video image, detecting targets in each video image, and fusing the extracted image features of each image and the corresponding intrinsic parameter matrix and extrinsic parameter matrix to generate a disparity map includes:

[0030] Inputting multiple video images and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the disparity estimation network, performing image feature extraction on each video image, and obtaining image features of each video image;

[0031] Detecting the target in each video frame based on the extracted image features of each video frame;

[0032] For each goal:

[0033] Perform feature matching on the extracted image features of each channel to obtain the image features of the same target under the perspective of each camera; and

[0034] Calculate the disparity value of the same target based on the image features of the same target under the perspective of each camera, as well as the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera;

[0035] A disparity map is generated based on the calculated disparity values ​​of each target.

[0036] Multiple video frames and the corresponding intrinsic parameter matrix and extrinsic parameter matrix are input into the disparity estimation network. The image features in each video frame are extracted and the targets in the corresponding video frames are detected based on the image features. For each detected target: the image features of the target in different video frames are compared and matched to determine the correspondence between the same features of the target under different viewing angles. Based on the matched feature correspondence and the intrinsic parameter matrix and extrinsic parameter matrix of the camera, the disparity value of the target is calculated, where the disparity value is used to characterize the position change of the same target in the video frames taken by different cameras. A disparity map is generated based on the calculated disparity values ​​of each target, where the disparity map provides a visual representation of the depth information for each target in each video frame, which is used to subsequently determine the three-dimensional coordinates of the target.

[0037] In one embodiment of the present invention, the disparity value of the same target is calculated based on the image features of the same target under different camera viewing angles, and the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera, including:

[0038] Identify and record the image coordinates of each image feature of the same target under the corresponding camera perspective;

[0039] According to the intrinsic parameter matrix of the camera, the image coordinates of each image feature are converted into the three-dimensional coordinates in the corresponding camera;

[0040] According to the camera's extrinsic parameter matrix, the three-dimensional coordinates of each image feature are converted into world coordinates in the world coordinate system;

[0041] Calculate the difference between the world coordinates of each image feature and use the calculated difference as the disparity value of the same target.

[0042] The same object is identified in video footage captured by cameras at different locations, and the position of the object's image features from each camera's perspective, i.e., the image coordinates, is recorded. Image coordinates represent the position of a pixel on a two-dimensional image plane and are typically composed of a horizontal coordinate (x) and a vertical coordinate (y). After obtaining the image coordinates of each image feature, the camera's intrinsic parameter matrix is ​​used to transform the image coordinates of the corresponding image feature from the two-dimensional image space to the camera's local three-dimensional coordinate system, taking into account factors such as the camera's focal length and optical center position. This results in the image feature's three-dimensional coordinates within the corresponding camera. The camera's extrinsic parameter matrix is ​​then used to uniformly transform the three-dimensional coordinates of the same image feature from all cameras into a world coordinate system, generating the image feature's world coordinates. This allows the camera to represent the position of the same object within a unified framework regardless of its position and orientation. In the world coordinate system, the difference in the coordinates of the same object in different world coordinate systems is calculated and used as the disparity value of the object. Since disparity refers to the difference in image coordinates of the same object from different perspectives, calculating disparity in the world coordinate system takes into account the actual positional differences of the object in the real world, rather than simply the visual differences on the image plane. This calculation can more realistically reflect the relative position and depth changes of objects in three-dimensional space.

[0043] S3. Input the disparity map, multi-channel video images and the corresponding intrinsic parameter matrix and extrinsic parameter matrix into the target tracking network of the multi-view target detection model, use the intrinsic parameter matrix and the extrinsic parameter matrix to perform spatial correction of the corresponding video images, obtain the two-dimensional annotation results of each target from the corrected video images, and generate the three-dimensional coordinates of each target based on the corrected video images and the disparity map.

[0044] The disparity map and video clips from different perspectives are input into the target tracking network. At the same time, each video frame is accompanied by a corresponding intrinsic parameter matrix and extrinsic parameter matrix. In the target tracking network, the corresponding video frame is geometrically distorted according to the intrinsic parameter matrix to ensure the correct position of the features in the video frame. The same target presented in different video frames is aligned to a world coordinate system according to the extrinsic parameter matrix to facilitate more accurate analysis of the content of the video frame. One or more preset targets are detected in the corrected video, and corresponding two-dimensional annotation results are generated for each detected target. By analyzing the two-dimensional annotation results and disparity map of the target in the corrected video frame, the position of the target in three-dimensional space can be determined, and the three-dimensional coordinates of the target can be generated.

[0045] In one embodiment of the present invention, the disparity map, multiple video frames, and corresponding intrinsic parameter matrices and extrinsic parameter matrices are input into the target tracking network of the multi-view target detection model, spatial correction of the corresponding video frames is performed using the intrinsic parameter matrix and the extrinsic parameter matrix, a two-dimensional annotation result of each target is obtained from the corrected video frames, and three-dimensional coordinates of each target are generated based on the corrected video frames and the disparity map, including:

[0046] Inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network, correcting the distortion of each video image based on the intrinsic parameter matrix and the extrinsic parameter matrix, and aligning each video image to a unified preset reference perspective;

[0047] Generating a corresponding two-dimensional annotation result for each target in the corrected video image; wherein the two-dimensional annotation result includes the attribute and size information of the corresponding target;

[0048] Based on the two-dimensional annotation results and disparity map of each target, the three-dimensional coordinates of each target are generated.

[0049] The disparity map and multiple video frames are fed into the target tracking network. Distortion correction is performed on each video frame using its intrinsic parameter matrix, correcting for geometric distortion introduced by the camera. The corrected video frames are then unified to a common viewing angle using an extrinsic parameter matrix. Two-dimensional annotations are generated for each pre-defined target in each corrected video frame. These 2D annotations include the target's position, attributes, and dimensions within the video frame. The 3D coordinates of the corresponding target are calculated by combining the 2D annotations with the disparity map.

[0050] In one embodiment of the present invention, the process of spatial correction of the video picture includes:

[0051] Based on the image distortion correction algorithm, the position of each pixel in the corresponding video image is adjusted according to the distortion coefficient in the intrinsic parameter matrix;

[0052] Perform perspective transformation on the adjusted video image based on the corresponding intrinsic parameter matrix;

[0053] The corresponding extrinsic matrix is ​​used to correct the position of the video image after perspective transformation.

[0054] Using an image distortion correction algorithm, distortion coefficients are read from an intrinsic parameter matrix. These coefficients describe the degree of image distortion. Distortion includes, but is not limited to, radial and tangential distortion. The corresponding distortion coefficients are read based on the type of distortion to be corrected. For each pixel in the video image, its correct position is calculated according to a preset mathematical equation based on its position in the image and the distortion coefficient. The image distortion correction algorithm remaps the pixel positions across the entire video image, reducing or eliminating the effects of distortion for a more realistic viewing experience. The intrinsic parameter matrix is ​​then extracted, and each pixel after distortion correction is transformed using the corresponding intrinsic parameter matrix. This perspective transformation ensures that objects in the image appear to have the correct depth and perspective proportions. Finally, by applying the corresponding extrinsic parameter matrix to each perspective-transformed video image, each point in the image is converted from the camera coordinate system to the world coordinate system, achieving position correction of the video image.

[0055] In one embodiment of the present invention, after obtaining the two-dimensional annotation results of each target in each video frame based on the corrected video frame and the disparity map, and generating the three-dimensional coordinates of each target, it also includes: displaying the two-dimensional annotation results of each target in the corresponding video frame.

[0056] After the calculation of the disparity map and the generation of three-dimensional coordinates based on it, the two-dimensional position of each target in the video image has been determined. These two-dimensional annotation results contain the target's location information (such as the coordinates of the bounding box) and possible other attribute information. By drawing a bounding box, adding a label description or other visual cues, the two-dimensional annotation results of each target (such as a bounding box, a label, etc.) are superimposed on the original video image or the corrected video image, so that the observer can intuitively identify and locate each target in the video, such as Figure 2 In addition, the 3D position data can be displayed alongside the video or through an interactive interface.

[0057] In another embodiment of the present invention, a training method for a multi-view object detection model is provided, the training method comprising:

[0058] Obtaining multiple synchronously collected video frames, as well as the corresponding two-dimensional annotation result set and three-dimensional coordinate set of each video frame set;

[0059] Input multiple video frames, their corresponding intrinsic and extrinsic matrixes into the disparity estimation network of the multi-view object detection model, extract the image feature set of each video frame, detect the object in each video frame, and fuse the extracted image feature set with the intrinsic and extrinsic matrix to generate a disparity atlas.

[0060] Inputting the disparity atlas, the multi-channel video frame set, the intrinsic parameter matrix, and the extrinsic parameter matrix into the target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video frame set using the intrinsic parameter matrix and the extrinsic parameter matrix, obtaining a two-dimensional annotation result of each target from the corrected video frame, and generating a predicted three-dimensional coordinate set for each target based on the corrected video frame set and the disparity atlas;

[0061] Based on the difference between the predicted two-dimensional annotation result set and the two-dimensional annotation result set, and the difference between the predicted three-dimensional coordinate set and the three-dimensional coordinate set, the parameters of the multi-view target detection model are updated to obtain a trained multi-view target detection model.

[0062] See Figure 3 The method collects multiple synchronously acquired video streams. Each video stream corresponds to a set of 2D annotations and 3D coordinate data. The 2D annotations contain the location and attribute information of each target in the video frame, while the 3D coordinates represent the position of each target in space. The multi-channel video streams and their corresponding intrinsic and extrinsic parameter matrices are fed into the disparity estimation network (i.e., layer L1). This extracts image features from each video stream and matches these features across multiple viewpoints to detect targets. The image features and intrinsic and extrinsic parameter matrix information from different camera viewpoints are combined to calculate and generate a disparity atlas. The disparity atlas and the extrinsic parameter matrices required for correction are then fed into the object tracking network (i.e., layer L2). Spatial correction is performed, geometrically and positionally correcting the multi-channel video streams to ensure that the same target observed from different viewpoints is correctly aligned in the image. The 2D annotations for each target are extracted from the corrected video frames and predicted 3D coordinates are calculated using the disparity atlas. The predicted 2D annotations and 3D coordinates are compared with the actual annotations and coordinate data to calculate the difference. The parameters of the multi-view target detection model are updated based on these differences, and the model is trained and optimized to obtain a trained multi-view target detection model.

[0063] Specifically, in the present invention, data annotation is explained using experimental videos as an example. In a laboratory environment, the camera in the examination room is fixed and the user operates on a specific laboratory table. In this setting, static objects such as the desktop and acquisition equipment become reference points for determining the spatial relationship between multiple cameras, which helps to simplify the annotation process. In the initial stage, these fixed objects are annotated once, and then the annotation information can be directly copied to avoid duplication of work. For dynamic objects used in the experiment, such as equipment, it is necessary to annotate them from the perspectives of multiple cameras. Assuming that two cameras are used, the annotation information of each object will include the coordinates and size from the two perspectives, as well as the circumscribed cube coordinates of the object in three-dimensional space. For example, the object obj1 is annotated as [x1, y1, w1, h1] under the first camera and as [x2, y2, w2, h2] under the second camera, where x1 and y1 are the starting coordinates of the target box of obj1 under the first camera, w1 and h1 are the width and height of the target box respectively, and x2 and y2 are the starting coordinates of the target box of obj1 under the second camera, and w2 and h2 are the width and height of the target box respectively. The coordinates of the circumscribed cube in three-dimensional space are represented as [px1,py1,pz1,px2,py2,pz2,px3,py3,pz3], where px1,py1,pz1 are the coordinates of the first point of the cube, px2,py2,pz2 are the coordinates of the second point, and px3,py3,pz3 are the coordinates of the third point. It should be noted that these three points of the cube are not coplanar. Furthermore, when labeling the data, it is necessary to ensure that each pair of images contains at least three fixed boundary matching points. These point annotations will be used in the subsequent intrinsic and extrinsic supervised learning process. To accurately reflect the experimental environment, the experimental table is set to zero height during labeling, corresponding to the z-axis; the table length is defined as the x-axis, with the center of the long side as the origin; and the table width corresponds to the y-axis, with its origin at the center of the long side. This coordinate system helps accurately map the spatial position of the experimental environment and equipment, enabling effective spatial analysis and tracking in a multi-camera system.

[0064] The labeled video image is input into the multi-view target detection model, where the input dimension of the model is [b, c, h, w], where b represents the batch size, c represents the number of channels, which is 6 here, corresponding to the top view and the other view, with each view occupying 3 channels; h and w represent the height and width of the image, respectively. In the present invention, it is assumed that all cameras used are of the same model, so their intrinsic parameter matrices are assumed to be consistent and can be initialized using predetermined parameters. The extrinsic parameter matrix is ​​initialized according to the specific camera layout. For example, a top view and a front view can be assumed to set the initial value of the extrinsic parameter matrix.

[0065] Combining the existing two-dimensional target detection loss function, the present invention incorporates the three-dimensional spatial output into the spatial transformation matrix constraint. The difference L of the multi-view target detection model is shown in formula (1):

[0066] L=L1+L2+L3(1)

[0067] Among them, L1 is the target loss of the first camera, L2 is the target loss of the second camera, and L3 is the spatial coordinate loss of the target. Specifically, the specific calculation of L1 is shown in formula (2):

[0068] L1=L pos1 +a1·L obj1 +a2·L cls1 (2)

[0069] Among them, L pos1 is the position loss, L obj1 is the target confidence loss, L cls1 is the category loss, a1 and a2 are the preset weight coefficients. Position loss L pos It is used to measure the difference between the position (coordinates) of the predicted bounding box and the true bounding box. The specific calculation method is shown in formula (3):

[0070]

[0071] Among them, IoU is the intersection over union (IoU) of the predicted box and the true box, p is the center point distance between the predicted box and the true box, C is the diagonal length of the minimum closure area that can cover both the predicted box and the true box, γ is a parameter to measure the consistency of the aspect ratio, and α is the weight coefficient.

[0072] Target loss L obj1 Used to measure the difference between the predicted target confidence and the actual target confidence. The target confidence indicates the probability of whether a bounding box contains an object. For a bounding box with a real target, the target confidence should be 1; for a bounding box with background or no target, the target confidence should be 0. Specifically, the target loss L obj1 The calculation of is shown in formula (4):

[0073] L obj =-∑ i y i logp i +(1-y i )log(1-p i ) (4)

[0074] Among them, y i is the true target confidence, which is 1 for the bounding box of the true target and 0 otherwise, p iis the target confidence of the prediction.

[0075] Class loss L cls1 Used to measure the difference between the predicted category probability and the true category. For each bounding box, the model predicts a category probability distribution. Specifically, the category loss is calculated as shown in formula (5):

[0076] L cls =-∑ i y i logp i (5)

[0077] Among them, y i is the true category probability distribution, usually a one-hot encoding vector, p i is the predicted class probability distribution.

[0078] It can be understood that the target loss of the second camera is L2 = L pos2 +a1·L obj2 +a2·L cls2 , where the position loss L pos2 , target loss L obj2 , category loss L cls2 The calculation is similar to the calculation of the first camera mentioned above and will not be repeated here. The spatial coordinate loss L3 of the target is obtained by calculating the sampling regression loss and the spatial intersection-over-union loss, as shown in formula (6):

[0079] L3=L Huber +(1-L cubic ) (6)

[0080] Among them, L Huber is the sampling regression loss, L cubic is the spatial intersection-over-union loss, and the sampling regression loss L Huber As shown in formula (7):

[0081]

[0082] Where y is the true value, y^ is the predicted value, and δ is a harmonic parameter used to balance the mean square error (MSE) when the predicted value is close to the true value and provide a linear loss growth when the predicted value is significantly different from the true value, thereby enhancing the robustness of the model. cubic It is defined as the intersection-over-union ratio, that is, Among them, V inter is the intersection volume of the predicted and true bounding cubes, V union is the union volume of the predicted and true bounding cubes. Specifically, V inter and V union The calculation of is shown in formula (8):

[0083]

[0084] in, To predict the coordinates of a corner point of the cube, is the coordinate of a corner point of the real cube, For the prediction cube and The coordinates of the corresponding diagonal points, For real cube and The coordinates of the corresponding diagonal points, To predict the coordinates of another corner point of the cube, and these three corner points are not coplanar, is the coordinate of another corner point of the real cube, and these three corner points are not coplanar. Furthermore, in the present invention, the difference between the calculated disparity map and the real disparity map can be added to the total disparity to make the final disparity map more accurate.

[0085] See Figure 4 The multi-view target detection system 100 includes: a data acquisition module 110, a disparity map generation module 120, and a target detection module 130. The data acquisition module 110 is used to synchronously capture multiple video images and obtain the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera; the disparity map generation module 120 is used to input the multiple video images and the corresponding intrinsic parameter matrix and extrinsic parameter matrix into the disparity estimation network of the multi-view target detection model, extract the image features of each video image, detect the target in each video image, and fuse the extracted image features of each image image and the corresponding intrinsic parameter matrix and extrinsic parameter matrix to generate a disparity map; the target detection module 130 is used to input the disparity map, multiple video images, and the corresponding intrinsic parameter matrix and extrinsic parameter matrix into the target tracking network of the multi-view target detection model, perform spatial correction of the corresponding video image using the intrinsic parameter matrix and extrinsic parameter matrix, obtain the two-dimensional annotation result of each target from the corrected video image, and generate the three-dimensional coordinates of each target based on the corrected video image and the disparity map.

[0086] The specific limitations of the multi-view object detection system can be found in the limitations of the multi-view object detection method described above and will not be further elaborated here. Each module in the multi-view object detection system described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the modules described above can be embedded in or independent of a processor in a computer device in hardware format, or stored in a memory in a computer device in software format, allowing the processor to invoke operations corresponding to each of the modules.

[0087] It should be noted that, in order to highlight the innovative part of the present invention, this embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in this embodiment.

[0088] See Figure 5 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a program for multi-view target detection.

[0089] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Furthermore, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code for multi-view target detection, etc., but can also be used to temporarily store data that has been output or is to be output.

[0090] In some embodiments, the processor 13 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1. It utilizes various interfaces and circuits to connect the various components of the entire electronic device 1. It executes or runs programs or modules stored in the memory 12 (such as a program for multi-view target detection) and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0091] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the multi-view target detection method.

[0092] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition module 110, a disparity map generation module 120, and an object detection module 130.

[0093] The above-mentioned integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, computer device, or network device, etc.) or a processor to perform part of the functions of the multi-view object detection method described in various embodiments of the present application.

[0094] In summary, the present invention discloses a multi-perspective target detection and model training method, system, device and medium, which adopts a two-dimensional detection network model and integrates binocular camera measurement technology to output the three-dimensional coordinates of the target from multi-camera data. Due to the simplicity of the model structure, it is easy to implement and maintain. In addition, in the present invention, images from multiple cameras are synchronously input and used for detection, realizing the recognition and association of the same target under different perspectives. This method enhances the utilization of spatial information and significantly improves the accuracy and reliability of detection compared to a single-camera system. Compared with traditional methods, this multi-camera target detection solution avoids dependence on large amounts of point cloud data and multi-scale scaling, making the model more lightweight. This feature makes it more practical and efficient in industrial applications, especially in situations where there are strict requirements on computing resources and processing speed. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0095] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A multi-view target detection method, characterized in that: The method comprises: Synchronously capture multiple video images and obtain the intrinsic and extrinsic parameter matrices of the corresponding cameras; Input multiple video images and the corresponding intrinsic and extrinsic parameter matrices into the disparity estimation network of the multi-view object detection model, extract the image features of each video image, detect the objects in each video image, and fuse the extracted image features and the corresponding intrinsic and extrinsic parameter matrices to generate a disparity map; Inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video images using the intrinsic parameter matrix and the extrinsic parameter matrix, obtaining a two-dimensional annotation result of each target from the corrected video images, and generating three-dimensional coordinates of each target based on the corrected video images and the disparity map; The method includes inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video images using the intrinsic parameter matrix and the extrinsic parameter matrix, obtaining a two-dimensional annotation result of each target from the corrected video images, and generating three-dimensional coordinates of each target based on the corrected video images and the disparity map. Inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network, correcting the distortion of each video image based on the intrinsic parameter matrix and the extrinsic parameter matrix, and aligning each video image to a unified preset reference perspective; Generating a corresponding two-dimensional annotation result for each target in the corrected video image; wherein the two-dimensional annotation result includes the attribute and size information of the corresponding target; Based on the two-dimensional annotation results and disparity map of each target, the three-dimensional coordinates of each target are generated.

2. The multi-view target detection method according to claim 1, wherein: The method includes inputting multiple video images and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the disparity estimation network of the multi-view target detection model, extracting image features of each video image, detecting targets in each video image, and fusing the extracted image features and corresponding intrinsic parameter matrices and extrinsic parameter matrices to generate a disparity map, including: Inputting multiple video images and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the disparity estimation network, performing image feature extraction on each video image, and obtaining image features of each video image; Detecting the target in each video frame based on the extracted image features of each video frame; For each goal: Perform feature matching on the extracted image features of each channel to obtain the image features of the same target under the perspective of each camera; and Calculate the disparity value of the same target based on the image features of the same target under the perspective of each camera, as well as the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera; A disparity map is generated based on the calculated disparity values ​​of each target.

3. The multi-view target detection method according to claim 2, characterized in that: The method of calculating the disparity value of the same target based on the image features of the same target under different camera viewing angles and the intrinsic parameter matrix and extrinsic parameter matrix of the corresponding camera includes: Identify and record the image coordinates of each image feature of the same target under the corresponding camera perspective; According to the intrinsic parameter matrix of the camera, the image coordinates of each image feature are converted into the three-dimensional coordinates in the corresponding camera; According to the camera's extrinsic parameter matrix, the three-dimensional coordinates of each image feature are converted into world coordinates in the world coordinate system; Calculate the difference between the world coordinates of each image feature and use the calculated difference as the disparity value of the same target.

4. The multi-view target detection method according to claim 1, wherein: The process of video picture space correction includes: Based on the image distortion correction algorithm, the position of each pixel in the corresponding video image is adjusted according to the distortion coefficient in the intrinsic parameter matrix; Perform perspective transformation on the adjusted video image based on the corresponding intrinsic parameter matrix; The corresponding extrinsic matrix is ​​used to correct the position of the video image after perspective transformation.

5. The multi-view target detection method according to claim 1, wherein: After obtaining the two-dimensional annotation result of each target in each video frame based on the corrected video frame and the disparity map and generating the three-dimensional coordinates of each target, the method further includes: displaying the two-dimensional annotation result of each target in the corresponding video frame.

6. A training method for a multi-view object detection model, characterized in that: The training method comprises: Obtaining multiple synchronously collected video frames, as well as the corresponding two-dimensional annotation result set and three-dimensional coordinate set of each video frame set; Input multiple video frames, their corresponding intrinsic and extrinsic matrixes into the disparity estimation network of the multi-view object detection model, extract the image feature set of each video frame, detect the object in each video frame, and fuse the extracted image feature set with the intrinsic and extrinsic matrix to generate a disparity atlas. Inputting the disparity atlas, the multi-channel video frame set, the intrinsic parameter matrix, and the extrinsic parameter matrix into the target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video frame set using the intrinsic parameter matrix and the extrinsic parameter matrix, obtaining a two-dimensional annotation result of each target from the corrected video frame, and generating a predicted three-dimensional coordinate set for each target based on the corrected video frame set and the disparity atlas; Based on the difference between the predicted two-dimensional annotation result set and the two-dimensional annotation result set, and the difference between the predicted three-dimensional coordinate set and the three-dimensional coordinate set, updating the parameters of the multi-view object detection model to obtain a trained multi-view object detection model; The method includes inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video images using the intrinsic parameter matrix and the extrinsic parameter matrix, obtaining a two-dimensional annotation result of each target from the corrected video images, and generating three-dimensional coordinates of each target based on the corrected video images and the disparity map. Inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network, correcting the distortion of each video image based on the intrinsic parameter matrix and the extrinsic parameter matrix, and aligning each video image to a unified preset reference perspective; Generating a corresponding two-dimensional annotation result for each target in the corrected video image; wherein the two-dimensional annotation result includes the attribute and size information of the corresponding target; Based on the two-dimensional annotation results and disparity map of each target, the three-dimensional coordinates of each target are generated.

7. A multi-view target detection system, characterized in that: The system comprises: The data acquisition module is used to synchronously capture multiple video images and obtain the intrinsic and extrinsic parameter matrices of the corresponding cameras; The disparity map generation module is used to input multiple video images and the corresponding intrinsic parameter matrix and extrinsic parameter matrix into the disparity estimation network of the multi-view object detection model, extract the image features of each video image, detect the target in each video image, and fuse the extracted image features of each video image with the corresponding intrinsic parameter matrix and extrinsic parameter matrix to generate a disparity map; a target detection module, configured to input the disparity map, multiple video frames, and corresponding intrinsic and extrinsic parameter matrices into a target tracking network of the multi-view target detection model, perform spatial correction of the corresponding video frames using the intrinsic and extrinsic parameter matrices, obtain a two-dimensional annotation result for each target from the corrected video frames, and generate three-dimensional coordinates of each target based on the corrected video frames and the disparity map; The method includes inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network of the multi-view target detection model, performing spatial correction of the corresponding video images using the intrinsic parameter matrix and the extrinsic parameter matrix, obtaining a two-dimensional annotation result of each target from the corrected video images, and generating three-dimensional coordinates of each target based on the corrected video images and the disparity map. Inputting the disparity map, multiple video images, and corresponding intrinsic parameter matrices and extrinsic parameter matrices into the target tracking network, correcting the distortion of each video image based on the intrinsic parameter matrix and the extrinsic parameter matrix, and aligning each video image to a unified preset reference perspective; Generating a corresponding two-dimensional annotation result for each target in the corrected video image; wherein the two-dimensional annotation result includes the attribute and size information of the corresponding target; Based on the two-dimensional annotation results and disparity map of each target, the three-dimensional coordinates of each target are generated.

8. An electronic device, characterized in that: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the multi-view target detection method according to any one of claims 1 to 5 or the multi-view target detection model training method according to claim 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the steps of the multi-view target detection method described in any one of claims 1 to 5 or the multi-view target detection model training method described in claim 6.

Images