Target tracking and display method and device for billion-pixel computational imaging system

Abstract

The present application relates to a target tracking and display method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product for a billion-pixel computational imaging system. The method comprises: obtaining a fused video based on each local video captured by an array camera; detecting and tracking the target object in each frame of the fused video image, and storing the fused position information of each target object in a target information table; sending the panoramic playback image and the global unique identifier and feature image of each target object to the client; receiving a target tracking and display request sent by the client, querying the matching fused position information in the target information table based on the global unique identifier of the specified tracking target and the timestamp information of the current frame of the fused video image, and obtaining the corresponding local fused image in the current frame of the fused video image and sending it to the client. The use of this method can improve the efficiency, operational convenience and accuracy of target tracking display.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target tracking and display method and apparatus for a billion-pixel computational imaging system, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. Background Art

[0002] Video surveillance has been widely used in the security field due to its advantages such as high reliability, timelyness, and ease of viewing. When the monitoring area is large, to achieve ultra-high-definition video surveillance of large scenes, billion-pixel computational imaging systems based on array cameras have emerged. These systems use multiple long-focus, narrow-field-of-view sub-cameras contained in the array to capture multiple HD videos of the monitored scene. These videos are then stitched together to create ultra-high-definition fused videos with a large or wide field of view, reaching up to billion-pixel levels. These fused videos are then displayed on display terminals and are suitable for video capture and security monitoring of large scenes such as airports, highways, parks, sports stadiums, border crossings, and sea surfaces.

[0003] In some scenarios, due to the resolution limitations of the display terminal, in order to display the full-frame fused video on the display terminal, it is usually necessary to reduce (downsample or downsample) the high-resolution fused video to an appropriate resolution for display. When the staff needs to track and view the movements and dynamics of a specified target (such as a person, car, etc.) in the fused video, they can select the local picture where the specified target is located and enlarge it for display.

[0004] However, as the designated target moves, if the above method is used for target tracking and display, new local images need to be manually selected frequently for zooming in, which results in low efficiency and poor operational convenience. Summary of the Invention

[0005] Based on this, it is necessary to provide a target tracking and display method, device, billion-pixel computational imaging system, computer equipment, computer-readable storage medium and computer program product that can improve efficiency and operational convenience in response to the above technical problems.

[0006] In a first aspect, the present application provides a target tracking and display method for a billion-pixel computational imaging system. The method comprises:

[0007] Acquire multiple channels of local videos captured by the array camera, and stitch and fuse the local videos to obtain a fused video;

[0008] Performing target detection and target tracking on each frame of the fused video image to obtain fused position information and feature images of each target object in the fused video image, and correspondingly storing the globally unique identifier of the target object, the timestamp information of the fused video image, and the fused position information in a target information table;

[0009] Acquire a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in a panoramic video playback area of ​​a first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in a target display area of ​​the first display interface;

[0010] receiving a target tracking display request sent by the client, and determining a globally unique identifier of a designated tracking target;

[0011] Based on the globally unique identifier of the designated tracking target and the timestamp information of the newly acquired current frame fused video image, the matching target fusion position information is queried in the target information table, and a local fusion image corresponding to the target fusion position information is obtained in the newly acquired current frame fused video image. The local fusion image is sent to the client, so that the client displays the local fusion image containing the designated tracking target in the target tracking playback area of ​​the second display interface.

[0012] In one embodiment, the performing target detection and target tracking on each frame of the fused video image to obtain fused position information and feature images of each target object in the fused video image includes:

[0013] For a group of local video images captured at the same time by each sub-camera included in the array camera, a target detection algorithm is used to respectively detect a target object contained in each of the local video images in the group, and obtain local position information of each target object in the group of local video images;

[0014] Performing coordinate transformation on the local position information of each target object in the local video image to obtain fused position information of each target object in the frame fused video image corresponding to the group of local video images;

[0015] Tracking each target object based on fusion position information of each target object in fusion video images of adjacent frames, and assigning a same global unique identifier to the same target object contained in fusion video images of different frames;

[0016] A feature image of each target object is obtained in the fused video image based on the fused position information.

[0017] In one embodiment, performing coordinate transformation on the local position information of each target object in the local video image to obtain the fused position information of each target object in the frame fused video image corresponding to the group of local video images includes:

[0018] Based on the coordinate transformation matrix corresponding to each of the sub-cameras, coordinate transformation is performed on the local position information of each target object in the local video image to obtain initial fused position information of each target object in the fused video image corresponding to the group of local video images;

[0019] Matching each target object in each of the partial video images with each target object in other partial video images in the same group to determine a successfully matched target object; the successfully matched target object refers to the same target object detected in at least two partial video images in the same group;

[0020] The multiple initial fused position information of the successfully matched target object are merged to obtain the fused position information of the target object, and the initial fused position information of the unsuccessfully matched target object is determined as the fused position information.

[0021] In one embodiment, matching each target object in each partial video image with each target object in other partial video images in the same group to determine a successfully matched target object includes:

[0022] Based on the initial fusion position information of each target object, the position overlap of any two target objects between any two local video images in the same group is calculated, and the target object with the position overlap greater than a preset threshold is determined as a successfully matched target object.

[0023] In one embodiment, the receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes:

[0024] Upon receiving a first type of target tracking display request sent by the client, obtaining a globally unique identifier of a designated tracking target carried in the target tracking display request; the first type of target tracking display request is a target tracking display request generated and sent to a server by the client based on the globally unique identifier of the designated tracking target corresponding to the designated feature image when the client detects a user's selection operation on a designated feature image among the feature images displayed in the target display area.

[0025] In one embodiment, after generating a panoramic playback image based on the fused video image of the current frame, the method further includes:

[0026] Determining the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and sending the timestamp information of the panoramic playback image to the client;

[0027] The receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes:

[0028] Upon receiving a second-type target tracking display request sent by the client, obtaining the specified timestamp information and the specified fusion position information carried in the target tracking display request; the second-type target tracking display request is when the client detects a target tracking display instruction triggered by a user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information based on the position of the triggering instruction, and generates and sends the target tracking display request to the server based on the specified timestamp information and the specified fusion position information;

[0029] The global unique identifier corresponding to the specified timestamp information and the specified fusion position information is searched in the target information table to obtain the global unique identifier of the specified tracking target.

[0030] In a second aspect, the present application also provides a target tracking and display system. The billion-pixel computational imaging system includes an array camera, a server, and a client, wherein the array camera includes multiple sub-cameras, wherein:

[0031] The array camera is used to capture a target scene through multiple sub-cameras and send the captured multiple local videos to a server;

[0032] The server is configured to stitch and fuse the acquired partial videos to obtain a fused video;

[0033] The server is further configured to perform target detection and target tracking on each frame of the fused video image, obtain fused position information and feature images of each target object in the fused video image, and store the globally unique identifier of the target object, the timestamp information of the fused video image, and the fused position information in a target information table; obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client;

[0034] The client is configured to receive the panoramic playback image and the globally unique identifier and feature image of each target object in the fused video image corresponding to the panoramic playback image sent by the server, display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and display the feature image corresponding to each target object in the target display area of ​​the first display interface;

[0035] The client is further configured to send a target tracking display request to the server in response to a user triggering operation on a designated tracking object;

[0036] The server is further configured to receive a target tracking display request sent by the client, determine a globally unique identifier of a designated tracking target, query matching target fusion position information in the target information table based on the globally unique identifier of the designated tracking target and the timestamp information of the newly acquired current-frame fusion video image, intercept a local fusion image corresponding to the target fusion position information in the new current-frame fusion video image, and send the local fusion image to the client;

[0037] The client is further configured to display the local fusion image including the designated tracking target in the target tracking playback area of ​​the second display interface.

[0038] In a third aspect, the present application further provides a target tracking and display device. The device comprises:

[0039] A fusion module is used to obtain multiple local videos captured by the array camera and stitch and fuse the local videos to obtain a fused video;

[0040] A target tracking module is used to perform target detection and target tracking on each frame of the fused video image, obtain the fused position information and feature image of each target object in the fused video image, and store the globally unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table;

[0041] The first sending module is used to obtain a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in the target display area of ​​the first display interface;

[0042] A determination module, configured to receive a target tracking display request sent by the client and determine a globally unique identifier of a designated tracking target;

[0043] The second sending module is used to query the matching target fusion position information in the target information table based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently obtained, and obtain the local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client, so that the client displays the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface.

[0044] In a fourth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0045] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0046] In a sixth aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0047] The above-mentioned target tracking and display method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product for the billion-pixel computational imaging system use an array camera to shoot and stitch together the target area to obtain multiple frames of fused video images, detect and track the target objects contained in each frame of the fused video image, obtain the fusion position information of each target object in each frame of the fused video image, and the characteristic image of each target object, send the panoramic playback image corresponding to the fused video image (such as a panoramic image matching the client resolution) to the client for display, and send the characteristic image and globally unique identification information of the target object contained in the fused video image to the client, so that the client can simultaneously display the panoramic playback image and the characteristic image of the target object contained in the image on the current interface. The user can trigger a target tracking display request on the current interface, such as by clicking on a feature image or clicking on the target's location in the panoramic playback image. After receiving the target tracking display request, the server queries the target information table for the location information of the specified tracking target in the current frame fused video image (usually the most recently fused frame fused video image) based on the global unique identifier of the specified tracking target for each subsequent moment in the current frame fused video image, and then intercepts the local fused image of the corresponding position in the current frame fused video image and sends it to the client for display.

[0048] Because the local fusion image primarily consists of the pixels of the designated tracking target and is much smaller than the original fused video image (typically smaller than a single frame of local video), it can match the client's display resolution without scaling down. Therefore, the local fusion image displayed on the client retains detailed information about the designated tracking target, meeting the needs of staff tracking and monitoring target dynamics. While watching the panoramic video, staff can trigger tracking display of any target in the current playback image at any time as needed. This convenient operation, efficient target tracking display, and guaranteed accuracy and continuity of target tracking display are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of an example of a billion-pixel computational imaging system;

[0050] Figure 2 1. A schematic flow chart of a target tracking and display method for a billion-pixel computational imaging system according to one embodiment;

[0051] Figure 3 is a schematic diagram of a first display interface in an example;

[0052] Figure 4 is a schematic diagram of a second display interface in an example;

[0053] Figure 5 A structural block diagram of a target tracking and display device for a billion-pixel computational imaging system in one embodiment;

[0054] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0056] First, before specifically introducing the technical solutions of the embodiments of this application, we will first introduce the technical background or technological evolution of the embodiments of this application. To achieve ultra-high-definition video surveillance of large scenes and wide fields of view, billion-pixel computational imaging systems based on array cameras have emerged. An array camera includes multiple sub-cameras with different shooting angles, each capturing a local high-definition video (e.g., 4K resolution) of the target scene. The fields of view of adjacent sub-cameras typically overlap. By stitching and fusing the local videos captured by each sub-camera, a large or wide field of view, ultra-high-resolution fused video (up to billion or billion pixels) can be generated for display or playback on a client. The client can be implemented using an electronic device including a display unit, specifically a large-screen ultra-high-definition spliced ​​display that matches the resolution of the fused video, ensuring that all captured details are clearly displayed.

[0057] The electronic device on the client can also be another device with a lower resolution. In this case, due to the limitations of the client's display resolution, in order to display the full field of view of the surveillance footage on the client, the high-resolution fused video image must typically be scaled down to an appropriate resolution for display. When personnel need to track and view the movement and status of a specific target (such as a person or vehicle) in the surveillance footage, they can select a portion of the footage containing the target for zoomed-in display. For example, by using a mouse or touchscreen to select the area where the target resides, the client can send the location information of the selected area to the server of the exascale pixel computational imaging system. The server can then extract the video data of the selected area from the fused video and send it to the client. Due to the small size of the selected area, the client can display the unscaled or slightly scaled-down footage containing detailed information about the target for the user to view. However, as the target moves, the user must continuously select new portions of the footage for zooming in, which is inconvenient and inefficient. Especially when the target moves rapidly, the inability to select the accurately positioned portion of the footage for zooming in will affect the continuity of target tracking and monitoring.

[0058] Based on this background, the applicant, through long-term research and development and experimental verification, has proposed the target tracking and display method of the present application, which can improve the efficiency of target tracking and monitoring based on billion-pixel fusion video, reduce operational difficulty, enhance user experience, and ensure the accuracy and continuity of target tracking display. In addition, it should be noted that the applicant has invested a lot of creative work in the discovery of the technical problem of this application and the technical solutions introduced in the following embodiments.

[0059] The target tracking and display method provided in the embodiment of the present application can be applied to Figure 1The illustrated exascale pixel computational imaging system includes an array camera 102 and a client 106 that communicate with a server 104 via a network. Client 106 can be implemented as an electronic device, including but not limited to various personal computers, laptops, smartphones, tablets, smart displays, IoT devices, portable wearable devices, and other electronic devices with a display unit. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0060] In one embodiment, Figure 2 As shown, a target tracking display method is provided, which can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0061] Step 202: Acquire multiple channels of local videos captured by the array camera, and merge the local videos to obtain a fused video.

[0062] In practice, a camera array can be used to capture video of a target area (such as an airport, park, sea surface, highway, bridge, sports field, etc.). The camera array can send the local video images captured by each sub-camera to a server, so that the server can obtain multiple local video images captured of the target area.

[0063] Then, the server can stitch and fuse a group of local video images taken by each sub-camera at the same time or quasi-same time (because the exposure time of each sub-camera is difficult to be absolutely consistent, there is a certain acceptable time difference between the shooting time of each local video image) to obtain the fused video image corresponding to that moment.

[0064] The stitching and fusion process can adopt the image stitching and fusion method in the related art. For example, based on the sample local video images captured by the array camera, feature point pairs can be searched based on the overlapping areas between each sample local video image, and then an image stitching and fusion model can be constructed based on the feature point pairs. Then, each group of local video images is stitched and fused using the image stitching and fusion model to obtain a fused video image. In an optional implementation, the array camera can also include a wide-angle camera, which captures a panoramic image of the target area (covering the entire target area or most of the target area) through the wide-angle camera to find feature point pairs for each sample local video image and the sample panoramic image, and construct a homography transformation matrix based on the feature point pairs so that each local video image is converted to the coordinate system of the panoramic image to obtain an image stitching and fusion model.

[0065] It is understood that the partial video images captured by the server can be real-time or pre-recorded. In some scenarios, such as when the field of view covered by all sub-cameras of the array camera is larger than the field of view of the target area to be captured, some sub-cameras can be controlled to capture partial video images as needed. Therefore, the server can only capture partial video images captured by some of the sub-cameras included in the array camera.

[0066] Step 204, target detection and target tracking are performed on each frame of the fused video image to obtain the fused position information and feature image of each target object in the fused video image, and the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information are correspondingly stored in the target information table.

[0067] Target objects can be preset types of objects, such as people, cars, drones, airplanes, and birds. You can pre-set the target type of interest and then use a target detection algorithm (such as YOLO, RCNN, or Fast-RCNN) to detect the target objects contained in the fused video image, obtaining the location information of each target object within the fused video image. A feature image is a partial image containing the features of the target object, captured from the fused video image or partial image where the target object was detected. A feature image can be a reduced image containing the entire target object or an image containing only partial features of the target object, such as a face or license plate.

[0068] Optionally, step 204 may specifically include the following steps:

[0069] Step 2041 : for a group of partial video images captured by each sub-camera at the same time, a target detection algorithm is used to detect the target object contained in each partial video image in the group, and obtain the local position information of each target object in the group of partial video images.

[0070] Step 2042 : performing coordinate conversion processing on the local position information of each target object in the local video image to obtain the fused position information of each target object in the frame of fused video image corresponding to the group of local video images.

[0071] Optionally, step 2042 specifically includes the following steps:

[0072] In step 20421, based on the coordinate transformation matrix (such as the homography transformation matrix) corresponding to each sub-camera, the local position information of each target object in the local video image is subjected to coordinate transformation processing to obtain the initial fused position information of each target object in the fused video image corresponding to the set of local video images.

[0073] In step 20422, each target object in each partial video image is matched with each target object in other partial video images in the same group to determine a successfully matched target object. A successfully matched target object refers to the same target object detected in at least two partial video images in the same group.

[0074] Optionally, the matching process includes: calculating the position overlap of any two target objects between any two local video images in the same group based on the initial fusion position information of each target object, and determining the target object whose position overlap is greater than a preset threshold as a successfully matched target object.

[0075] In step 20423, multiple initial fused position information of successfully matched target objects are merged to obtain the fused position information of the target object, and the initial fused position information of unsuccessfully matched target objects is determined as the fused position information.

[0076] Since the same target object may be captured simultaneously by adjacent sub-cameras, the position overlap of the initial fused position information is used to determine whether it is the same target. The initial fused position information of the same target is then merged (detection frame merging) to improve the detection accuracy of each target object in the fused image.

[0077] Step 2043 : Track each target object based on the fusion position information of each target object in the fused video images of adjacent frames, and assign the same globally unique identifier to the same target object contained in the fused video images of different frames.

[0078] Step 2044 : capturing a feature image of each target object in the fused video image based on the fused position information.

[0079] In practice, for non-merged target objects, a feature image can be captured from a partial image, and for merged target objects, a feature image can be captured from a fused image. Optionally, the captured image can be scaled down to a preset smaller resolution to serve as the feature image for easier display on the client.

[0080] Step 206 , obtaining the current frame fused video image, generating a panoramic playback image based on the current frame fused video image, and sending the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client.

[0081] The panoramic playback image is used by the client to display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and the characteristic image is used by the client to display the characteristic image corresponding to each target object in the target display area of ​​the first display interface. Figure 3The first display interface shown can display a panoramic playback image in the panoramic video playback area 301 and display a feature image corresponding to each target object in the target display area 302 .

[0082] Step 208: Receive the target tracking display request sent by the client, and determine the globally unique identifier of the designated tracking target.

[0083] Optionally, the target tracking display request includes a first type. A user can select a characteristic image of a designated target object in target display area 302, triggering the first type of target tracking display request. The first type of target tracking display request is generated and sent to the server by the client after detecting the user's selection of a designated characteristic image among the characteristic images displayed in the target display area, based on the globally unique identifier of the designated target object corresponding to the designated characteristic image. Upon receiving the first type of target tracking display request from the client, the server can obtain the globally unique identifier of the designated tracking target carried in the target tracking display request.

[0084] Optionally, step 206 also includes determining the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and sending the timestamp information of the panoramic playback image to the client. The target tracking display request includes a second type. The user can select a specified target object in the panoramic video playback area 301 to trigger a second type of target tracking display request. The second type of target tracking display request is that the client detects a target tracking display instruction triggered by the user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information according to the position of the trigger instruction, and generates and sends a target tracking display request to the server based on the specified timestamp information and the specified fusion position information.

[0085] When the server receives the second type of target tracking display request sent by the client, it can obtain the specified timestamp information and specified fusion position information carried in the target tracking display request, and query the target information table for the globally unique identifier corresponding to the specified timestamp information and the specified fusion position information to obtain the globally unique identifier of the specified tracking target.

[0086] Step 210, based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fused video image currently obtained, query the matching target fusion position information in the target information table, obtain the local fusion image corresponding to the target fusion position information in the new current frame fused video image, and send the local fusion image to the client.

[0087] The local fusion image is used by the client to display the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface. Figure 4 The second display interface shown includes a target tracking playback area 401 for tracking and displaying a local fusion image including a designated tracking target.

[0088] In the above-mentioned target tracking display method, a client can simultaneously display a panoramic playback image and a feature image of the target object contained in the image on the current interface. A user can trigger a target tracking display request on the current interface, such as by clicking on the feature image or clicking on the target's location in the panoramic playback image. Upon receiving the target tracking display request, the server queries the target information table for the location of the specified target in the current fused video frame (typically the most recently fused frame) based on the globally unique identifier of the specified target. The server then extracts a local fused image of the corresponding location in the current fused video frame and sends it to the client for display. Because the local fused image primarily contains pixels of the specified target and is much smaller than the original fused video image (typically smaller than the size of a local video frame), it can match the client's display resolution without compression. Therefore, the local fused image displayed by the client retains detailed information about the specified target, meeting the needs of staff members who need to track and view the target's dynamics. While watching the panoramic video, staff members can trigger tracking display of any target in the current playback image at any time as needed. This convenient operation, efficient target tracking display, and guaranteed accuracy and continuity of target tracking display are achieved.

[0089] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0090] Based on the same inventive concept, embodiments of the present application also provide a target tracking and display device for implementing the target tracking and display method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more target tracking and display device embodiments provided below can be found in the limitations of the target tracking and display method described above and will not be further elaborated here.

[0091] In one embodiment, Figure 5 As shown, a target tracking and display device 500 for a billion-pixel computational imaging system is provided, comprising: a fusion module 501, a target tracking module 502, a first sending module 503, a determination module 504, and a second sending module 505, wherein:

[0092] The fusion module 501 is used to obtain multiple local videos captured by the array camera, and to splice and fuse the local videos to obtain a fused video.

[0093] The target tracking module 502 is used to perform target detection and target tracking on each frame of the fused video image, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in the target information table.

[0094] The first sending module 503 is used to obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in the target display area of ​​the first display interface.

[0095] The determination module 504 is configured to receive the target tracking display request sent by the client and determine the globally unique identifier of the designated tracking target.

[0096] The second sending module 505 is used to query the matching target fusion position information in the target information table based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently obtained, and obtain the local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client, so that the client can display the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface.

[0097] In one embodiment, the target tracking module 502 is specifically used to: for a group of local video images taken by each of the sub-cameras at the same time, use a target detection algorithm to respectively detect the target object contained in each of the local video images in the group, and obtain the local position information of each target object in the group of local video images; perform coordinate conversion processing on the local position information of each target object in the local video image to obtain the fusion position information of each target object in the frame fusion video image corresponding to the group of local video images; track each target object based on the fusion position information of each target object in the fusion video images of adjacent frames, and assign the same global unique identifier to the same target object contained in the fusion video images of different frames; and intercept the feature image of each target object in the fusion video image based on the fusion position information.

[0098] In one embodiment, the target tracking module 502 is specifically used to: perform coordinate conversion processing on the local position information of each target object in the local video image based on the coordinate conversion matrix corresponding to each sub-camera to obtain the initial fusion position information of each target object in the fusion video image corresponding to the group of local video images; match each target object in each of the local video images with each target object in other local video images in the same group to determine the successfully matched target object; the successfully matched target object refers to the same target object detected in at least two local video images in the same group; merge multiple initial fusion position information of the successfully matched target object to obtain the fusion position information of the target object, and determine the initial fusion position information of the unmatched target object as the fusion position information.

[0099] In one embodiment, the target tracking module 502 is specifically used to: calculate the position overlap of any two target objects between any two local video images in the same group based on the initial fusion position information of each target object, and determine the target object whose position overlap is greater than a preset threshold as a successfully matched target object.

[0100] In one embodiment, the determination module 504 is specifically used to: upon receiving a first type of target tracking display request sent by the client, obtain the globally unique identifier of the specified tracking target carried in the target tracking display request; the first type of target tracking display request is a target tracking display request generated and sent to the server based on the globally unique identifier of the specified target object corresponding to the specified feature image, when the client detects a user's selection operation on a specified feature image among the feature images displayed in the target display area.

[0101] In one embodiment, the device further includes a third sending module for determining the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and sending the timestamp information of the panoramic playback image to the client. The determination module 504 is specifically used to: upon receiving the second type of target tracking display request sent by the client, obtain the specified timestamp information and specified fusion position information carried by the target tracking display request; the second type of target tracking display request is that the client detects a target tracking display instruction triggered by a user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information according to the position of the trigger instruction, generates and sends a target tracking display request to the server based on the specified timestamp information and the specified fusion position information; queries the target information table for a globally unique identifier corresponding to the specified timestamp information and the specified fusion position information to obtain a globally unique identifier for the specified tracking target.

[0102] Each module in the target tracking and display device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0103] The present application also provides a billion-pixel computational imaging system, including an array camera, a server, and a client. The array camera includes multiple sub-cameras, wherein:

[0104] The array camera is used to capture a target scene through multiple sub-cameras and send the captured multiple local videos to the server;

[0105] The server is configured to stitch and fuse the acquired partial videos to obtain a fused video;

[0106] The server is further configured to perform target detection and target tracking on each frame of the fused video image, obtain fused position information and feature images of each target object in the fused video image, and store the globally unique identifier of the target object, the timestamp information of the fused video image, and the fused position information in a target information table; obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client;

[0107] The client is configured to receive the panoramic playback image and the globally unique identifier and feature image of each target object in the fused video image corresponding to the panoramic playback image sent by the server, display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and display the feature image corresponding to each target object in the target display area of ​​the first display interface;

[0108] The client is further configured to send a target tracking display request to the server in response to a user triggering operation on a designated tracking object;

[0109] The server is further configured to receive a target tracking display request sent by the client, determine a globally unique identifier of a designated tracking target, query matching target fusion position information in the target information table based on the globally unique identifier of the designated tracking target and the timestamp information of the newly acquired current-frame fusion video image, intercept a local fusion image corresponding to the target fusion position information in the new current-frame fusion video image, and send the local fusion image to the client;

[0110] The client is further configured to display the local fusion image including the designated tracking target in the target tracking playback area of ​​the second display interface.

[0111] In one embodiment, the server is further used to: for a group of local video images taken by each of the sub-cameras at the same time, use a target detection algorithm to respectively detect the target object contained in each of the local video images in the group, and obtain the local position information of each target object in the group of local video images; perform coordinate conversion processing on the local position information of each target object in the local video image to obtain the fusion position information of each target object in the frame fusion video image corresponding to the group of local video images; track each target object based on the fusion position information of each target object in the fusion video images of adjacent frames, and assign the same globally unique identifier to the same target object contained in the fusion video images of different frames; and capture the feature image of each target object in the fusion video image based on the fusion position information.

[0112] In one embodiment, the server is further used to: perform coordinate transformation processing on the local position information of each target object in the local video image based on the coordinate transformation matrix corresponding to each sub-camera to obtain the initial fusion position information of each target object in the fusion video image corresponding to the group of local video images; match each target object in each of the local video images with each target object in other local video images in the same group to determine the successfully matched target object; the successfully matched target object refers to the same target object detected in at least two local video images in the same group; merge multiple initial fusion position information of the successfully matched target object to obtain the fusion position information of the target object, and determine the initial fusion position information of the unmatched target object as the fusion position information.

[0113] In one embodiment, the server is further used to: calculate the position overlap of any two target objects between any two local video images in the same group based on the initial fusion position information of each target object, and determine the target object whose position overlap is greater than a preset threshold as a successfully matched target object.

[0114] In one embodiment, receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes:

[0115] The client is also used to detect the user's selection operation on a specified feature image among the feature images displayed in the target display area, determine the globally unique identifier of the specified target object corresponding to the specified feature image, and send a first type target tracking display request carrying the globally unique identifier of the specified target object to the server.

[0116] The server is further configured to receive a first type target tracking display request sent by the client, and obtain a globally unique identifier of a designated tracking target carried in the target tracking display request.

[0117] In one embodiment, the server is further configured to determine the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and send the timestamp information of the panoramic playback image to the client.

[0118] The client is also used to detect a target tracking display instruction triggered by a user at any position in the panoramic video playback area, determine the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determine the specified fusion position information according to the position of the triggering instruction, and send a second type of target tracking display request containing the specified timestamp information and the specified fusion position information to the server.

[0119] The server is further used to receive a second type of target tracking display request sent by the client, obtain the specified timestamp information and specified fusion position information carried by the target tracking display request; query the target information table for a globally unique identifier corresponding to the specified timestamp information and the specified fusion position information, and obtain the globally unique identifier of the specified tracking target.

[0120] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required for or generated by executing the above-mentioned target tracking and display method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a target tracking and display method is implemented.

[0121] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0122] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0123] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0124] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0126] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0127] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the scope of the present application, and all such modifications and improvements fall within the scope of protection of the present application.

Claims

1. A target tracking and display method for a billion-pixel computational imaging system, characterized in that: The method comprises: Acquire multiple channels of local videos captured by the array camera, and stitch and fuse the local videos to obtain a fused video; Performing target detection and target tracking on each frame of the fused video image to obtain fused position information and feature images of each target object in the fused video image, and correspondingly storing the globally unique identifier of the target object, the timestamp information of the fused video image, and the fused position information in a target information table; Acquire a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in a panoramic video playback area of ​​a first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in a target display area of ​​the first display interface; receiving a target tracking display request sent by the client, and determining a globally unique identifier of a designated tracking target; Based on the globally unique identifier of the designated tracking target and the timestamp information of the newly acquired current-frame fused video image, searching the target information table for matching target fusion position information, obtaining a local fusion image corresponding to the target fusion position information in the newly acquired current-frame fused video image, and sending the local fusion image to the client, so that the client displays the local fusion image containing the designated tracking target in the target tracking playback area of ​​the second display interface; The performing of target detection and target tracking on each frame of the fused video image to obtain fused position information and feature images of each target object in the fused video image includes: For a group of local video images captured at the same time by each sub-camera included in the array camera, a target detection algorithm is used to respectively detect a target object contained in each of the local video images in the group, and obtain local position information of each target object in the group of local video images; Performing coordinate transformation on the local position information of each target object in the local video image to obtain fused position information of each target object in the frame fused video image corresponding to the group of local video images; Tracking each target object based on fusion position information of each target object in fusion video images of adjacent frames, and assigning a same global unique identifier to the same target object contained in fusion video images of different frames; A feature image of each target object is obtained in the fused video image based on the fused position information.

2. The method according to claim 1, characterized in that The coordinate transformation of the local position information of each target object in the local video image to obtain the fused position information of each target object in the frame fused video image corresponding to the group of local video images includes: Based on the coordinate transformation matrix corresponding to each of the sub-cameras, coordinate transformation is performed on the local position information of each target object in the local video image to obtain initial fused position information of each target object in the fused video image corresponding to the group of local video images; Matching each target object in each of the partial video images with each target object in other partial video images in the same group to determine a successfully matched target object; the successfully matched target object refers to the same target object detected in at least two partial video images in the same group; The multiple initial fused position information of the successfully matched target object are merged to obtain the fused position information of the target object, and the initial fused position information of the unsuccessfully matched target object is determined as the fused position information.

3. The method according to claim 2, characterized in that The matching process of each target object in each partial video image with each target object in other partial video images in the same group to determine a successfully matched target object includes: Based on the initial fusion position information of each target object, the position overlap of any two target objects between any two local video images in the same group is calculated, and the target object with the position overlap greater than a preset threshold is determined as a successfully matched target object.

4. The method according to claim 1, wherein The receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes: Upon receiving a first type of target tracking display request sent by the client, obtaining a globally unique identifier of a designated tracking target carried in the target tracking display request; the first type of target tracking display request is a target tracking display request generated and sent to a server by the client based on the globally unique identifier of the designated tracking target corresponding to the designated feature image when the client detects a user's selection operation on a designated feature image among the feature images displayed in the target display area.

5. The method according to claim 1, wherein After generating a panoramic playback image based on the current frame fusion video image, the method further includes: Determining the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and sending the timestamp information of the panoramic playback image to the client; The receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes: Upon receiving a second-type target tracking display request sent by the client, obtaining the specified timestamp information and the specified fusion position information carried in the target tracking display request; the second-type target tracking display request is when the client detects a target tracking display instruction triggered by a user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information based on the position of the triggering instruction, and generates and sends the target tracking display request to the server based on the specified timestamp information and the specified fusion position information; The global unique identifier corresponding to the specified timestamp information and the specified fusion position information is searched in the target information table to obtain the global unique identifier of the specified tracking target.

6. A billion-pixel computational imaging system, characterized in that: The billion-pixel computational imaging system includes an array camera, a server, and a client. The array camera includes multiple sub-cameras, wherein: The array camera is used to capture a target scene through multiple sub-cameras and send the captured multiple local videos to a server; The server is configured to stitch and fuse the acquired partial videos to obtain a fused video; The server is also used to perform target detection and target tracking on each frame of the fused video image of the fused video, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table; obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the global unique identifier and feature image of each target object in the current frame fused video image to the client; wherein, the target detection and target tracking on each frame of the fused video image of the fused video, and the fused position information and feature image of each target object in the fused video image, include : For a group of local video images captured at the same time by each sub-camera included in the array camera, a target detection algorithm is used to respectively detect the target object contained in each local video image in the group, and obtain the local position information of each target object in the group of local video images; the local position information of each target object in the local video image is coordinate-converted to obtain the fused position information of each target object in the frame fused video image corresponding to the group of local video images; based on the fused position information of each target object in the fused video images of adjacent frames, each target object is tracked, and the same globally unique identifier is assigned to the same target object contained in the fused video images of different frames; based on the fused position information, a feature image of each target object is obtained in the fused video image; The client is configured to receive the panoramic playback image and the globally unique identifier and feature image of each target object in the fused video image corresponding to the panoramic playback image sent by the server, display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and display the feature image corresponding to each target object in the target display area of ​​the first display interface; The client is further configured to send a target tracking display request to the server in response to a user triggering operation on a designated tracking object; The server is further configured to receive a target tracking display request sent by the client, determine a globally unique identifier of a designated tracking target, query matching target fusion position information in the target information table based on the globally unique identifier of the designated tracking target and the timestamp information of the newly acquired current-frame fusion video image, intercept a local fusion image corresponding to the target fusion position information in the new current-frame fusion video image, and send the local fusion image to the client; The client is further configured to display the local fusion image including the designated tracking target in the target tracking playback area of ​​the second display interface.

7. A target tracking display device, characterized in that: The device comprises: A fusion module is used to obtain multiple local videos captured by the array camera and stitch and fuse the local videos to obtain a fused video; The target tracking module is used to perform target detection and target tracking on each frame of the fused video image, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in the target information table; wherein, the target detection and target tracking on each frame of the fused video image, obtain the fused position information and feature image of each target object in the fused video image, includes: for a group of local video images taken at the same time by each sub-camera included in the array camera, using the target detection algorithm The method comprises the following steps: detecting the target objects contained in each of the partial video images in the group, obtaining the local position information of each target object in the group of partial video images; performing coordinate transformation on the local position information of each target object in the partial video images, obtaining the fusion position information of each target object in the frame fusion video image corresponding to the group of partial video images; tracking each target object based on the fusion position information of each target object in the fusion video images of adjacent frames, assigning the same global unique identifier to the same target object contained in the fusion video images of different frames; and obtaining the feature image of each target object in the fusion video image based on the fusion position information; The first sending module is used to obtain a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in the target display area of ​​the first display interface; A determination module, configured to receive a target tracking display request sent by the client and determine a globally unique identifier of a designated tracking target; The second sending module is used to query the matching target fusion position information in the target information table based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently obtained, and to intercept the local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client, so that the client can display the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images