Target tracking display method and device for hundred million pixel calculation imaging system

Abstract

The invention relates to a target tracking display method and device for a hundred million-level pixel calculation imaging system, the hundred million-level pixel calculation imaging system, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining a fused video according to each local video shot by an array camera; detecting and tracking a target object in each frame of fused video image, and correspondingly storing fused position information of each target object in a target information table; sending the panoramic playing image and the global unique identifier and the feature image of each target object to the client; and receiving a target tracking display request sent by the client, querying matched fusion position information in the target information table based on the global unique identifier of the specified tracking target and the timestamp information of the current frame of fusion video image, and obtaining a corresponding local fusion image in the current frame of fusion video image and sending the local fusion image to the client. By adopting the method, the efficiency, the operation convenience and the accuracy of target tracking display can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target tracking and display method and device for a billion-pixel computational imaging system, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. Background Art

[0002] Video surveillance has been widely used in the security field due to its advantages of high reliability, strong timeliness, and easy viewing. When the monitoring range is large, in order to achieve ultra-high-definition video monitoring of large scenes, a billion-pixel computational imaging system based on array cameras has emerged. The array camera contains multiple long-focus and narrow-field-of-view sub-cameras to shoot multiple local high-definition videos of the monitoring scene, and then the local videos are spliced ​​and fused to obtain ultra-high-definition fused videos with large or wide fields of view up to billion pixels, which are used for display terminals. It is suitable for video shooting and security monitoring of large scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.

[0003] In some scenarios, due to the resolution limitation of the display terminal, in order to display the full-frame fused video on the display terminal, it is usually necessary to reduce (downsample or downsample) the high-resolution fused video to an appropriate resolution for display. When the staff needs to track and view the movements and dynamics of specified targets (such as people, cars, etc.) in the fused video, they can select the local picture where the specified target is located for enlarged display.

[0004] However, as the designated target moves, if the above method is used for target tracking and display, new local images need to be frequently manually selected for zooming in, which results in low efficiency and poor operational convenience. Summary of the invention

[0005] Based on this, it is necessary to provide a target tracking and display method, device, billion-pixel computational imaging system, computer equipment, computer-readable storage medium and computer program product that can improve efficiency and operational convenience in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a target tracking and display method for a billion-pixel computational imaging system. The method comprises:

[0007] Acquire multiple local videos captured by the array camera, and splice and fuse the local videos to obtain a fused video;

[0008] Performing target detection and target tracking on each frame of the fused video image of the fused video to obtain fused position information and feature images of each target object in the fused video image, and correspondingly storing the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table;

[0009] Acquire a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in a panoramic video playback area of ​​a first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in a target display area of ​​the first display interface;

[0010] Receiving a target tracking display request sent by the client, and determining a globally unique identifier of a designated tracking target;

[0011] Based on the globally unique identifier of the designated tracking target and the timestamp information of the new current frame fused video image currently obtained, the matching target fusion position information is queried in the target information table, and a local fusion image corresponding to the target fusion position information is obtained in the new current frame fusion video image, and the local fusion image is sent to the client, so that the client displays the local fusion image containing the designated tracking target in the target tracking playback area of ​​the second display interface.

[0012] In one embodiment, the method according to claim 1 is characterized in that the performing target detection and target tracking on each frame of the fused video image to obtain the fused position information and feature image of each target object in the fused video image includes:

[0013] For a group of local video images taken by each sub-camera included in the array camera at the same time, a target detection algorithm is used to respectively detect the target object included in each local video image in the group, and obtain the local position information of each target object in the group of local video images;

[0014] Performing coordinate transformation on the local position information of each target object in the local video image to obtain fused position information of each target object in the frame fused video image corresponding to the group of local video images;

[0015] Based on the fusion position information of each target object in the fused video images of adjacent frames, each target object is tracked, and a same global unique identifier is assigned to the same target object contained in the fused video images of different frames;

[0016] A feature image of each of the target objects is obtained in the fused video image based on the fused position information.

[0017] In one embodiment, the coordinate transformation of the local position information of each target object in the local video image to obtain the fused position information of each target object in the frame fused video image corresponding to the group of local video images includes:

[0018] Based on the coordinate transformation matrix corresponding to each of the sub-cameras, coordinate transformation is performed on the local position information of each target object in the local video image to obtain the initial fused position information of each target object in the fused video image corresponding to the group of local video images;

[0019] Matching each target object in each of the local video images with each target object in other local video images in the same group to determine a successfully matched target object; the successfully matched target object refers to the same target object detected in at least two local video images in the same group;

[0020] The multiple initial fused position information of the successfully matched target object are merged to obtain the fused position information of the target object, and the initial fused position information of the unsuccessfully matched target object is determined as the fused position information.

[0021] In one embodiment, matching each target object in each of the partial video images with each target object in other partial video images in the same group to determine the successfully matched target object includes:

[0022] Based on the initial fusion position information of each target object, the position overlap of any two target objects between any two local video images in the same group is calculated, and the target object whose position overlap is greater than a preset threshold is determined as a successfully matched target object.

[0023] In one of the embodiments, the receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes:

[0024] Upon receiving a first type of target tracking display request sent by the client, a globally unique identifier of a designated tracking target carried in the target tracking display request is obtained; the first type of target tracking display request is a target tracking display request generated and sent to a server based on the globally unique identifier of a designated tracking target corresponding to the designated feature image, when the client detects a user's selection operation on a designated feature image among the feature images displayed in the target display area.

[0025] In one of the embodiments, after generating a panoramic playback image based on the current frame fused video image, the method further includes:

[0026] Determine the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and send the timestamp information of the panoramic playback image to the client;

[0027] The receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes:

[0028] In the case of receiving the second type of target tracking display request sent by the client, obtaining the specified timestamp information and the specified fusion position information carried by the target tracking display request; the second type of target tracking display request is that the client detects a target tracking display instruction triggered by a user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information according to the position of the triggering instruction, and generates and sends the target tracking display request to the server based on the specified timestamp information and the specified fusion position information;

[0029] The global unique identifier corresponding to the specified timestamp information and the specified fusion position information is searched in the target information table to obtain the global unique identifier of the specified tracking target.

[0030] In a second aspect, the present application also provides a target tracking and display system. The billion-pixel computational imaging system includes an array camera, a server, and a client, wherein the array camera includes a plurality of sub-cameras, wherein:

[0031] The array camera is used to shoot a video of a target scene through multiple sub-cameras, and send the shot multiple local videos to a server;

[0032] The server is used for splicing and fusing the acquired partial videos to obtain a fused video;

[0033] The server is also used to perform target detection and target tracking on each frame of the fused video image of the fused video, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table; obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the global unique identifier and feature image of each target object in the current frame fused video image to the client;

[0034] The client is used to receive the panoramic playback image sent by the server and the globally unique identifier and feature image of each target object in the fused video image corresponding to the panoramic playback image, display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and display the feature image corresponding to each target object in the target display area of ​​the first display interface;

[0035] The client is also used to send a target tracking display request to the server in response to a user's trigger operation on a designated tracking object;

[0036] The server is further configured to receive a target tracking display request sent by the client, determine a globally unique identifier of a specified tracking target, query matching target fusion position information in the target information table based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently acquired, intercept a local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client;

[0037] The client is further configured to display the local fusion image including the designated tracking target in the target tracking playback area of ​​the second display interface.

[0038] In a third aspect, the present application also provides a target tracking display device. The device comprises:

[0039] A fusion module is used to obtain multiple local videos captured by the array camera, and to splice and fuse the local videos to obtain a fused video;

[0040] A target tracking module is used to perform target detection and target tracking on each frame of the fused video image, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table;

[0041] A first sending module is used to obtain a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to a client; the panoramic playback image is used by the client to display the panoramic playback image in a panoramic video playback area of ​​a first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in a target display area of ​​the first display interface;

[0042] A determination module, configured to receive a target tracking display request sent by the client and determine a globally unique identifier of a specified tracking target;

[0043] The second sending module is used to query the matching target fusion position information in the target information table based on the global unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently obtained, and obtain the local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client, so that the client can display the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface.

[0044] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0045] In a fifth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0046] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0047] The above-mentioned target tracking and display method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product for a billion-pixel computational imaging system obtain multiple frames of fused video images by capturing and stitching the target area with an array camera, detect and track the target object contained in each frame of the fused video image, obtain the fusion position information of each target object in each frame of the fused video image, and the feature image of each target object, send the panoramic playback image corresponding to the fused video image (such as a panoramic image matching the client resolution) to the client for display, and send the feature image and globally unique identification information of the target object contained in the fused video image to the client, so that the client can simultaneously display the panoramic playback image and the feature image of the target object contained in the image on the current interface. The user can trigger a target tracking display request in the current interface, such as by clicking on a feature image or clicking on the location of the target in the panoramic playback image. After receiving the target tracking display request, the server queries the target information table for the location information of the specified tracking target in the current frame fused video image (usually the latest fused frame of the fused video image) for the current frame fused video image at each subsequent moment according to the global unique identifier of the specified tracking target, and then captures the local fused image of the corresponding position in the current frame fused video image and sends it to the client for display.

[0048] Since the local fusion image mainly contains the pixels of the specified tracking target, the size is much smaller than the size of the original fusion video image (usually smaller than the size of a frame of local video image), and can match the display resolution of the client without being reduced, the local fusion image displayed by the client can retain the detailed information of the specified tracking target to meet the needs of the staff to track and monitor the dynamics of the target. When watching the panoramic video, the staff can trigger the tracking display of any target in the current playback screen at any time as needed. The operation is convenient, the target tracking display is efficient, and the accuracy and continuity of the target tracking display can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a billion-pixel computational imaging system in an example;

[0050] Figure 2 A schematic diagram of a flow chart of a target tracking and display method for a billion-pixel computational imaging system in one embodiment;

[0051] Figure 3 is a schematic diagram of a first display interface in an example;

[0052] Figure 4 is a schematic diagram of a second display interface in an example;

[0053] Figure 5 A structural block diagram of a target tracking and display device for a billion-pixel computational imaging system in one embodiment;

[0054] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] First of all, before specifically introducing the technical solution of the embodiment of the present application, the technical background or technical evolution context on which the embodiment of the present application is based is introduced. In order to achieve ultra-high-definition video surveillance with large scenes and wide fields of view, a billion-pixel computational imaging system based on array cameras has emerged. The array camera includes multiple sub-cameras with different shooting angles, which respectively shoot local high-definition videos (such as 4K resolution) of the target scene. Usually, there is an overlapping area in the field of view of adjacent sub-cameras. By splicing and fusing the local videos shot by each sub-camera, a large or wide field of view, ultra-high-resolution fused video (up to 100 million or 1 billion pixels) can be obtained for display or playback on the client. The client can be implemented by an electronic device including a display unit, specifically an ultra-clear spliced ​​large-screen display that matches the resolution of the fused video, so that all the detailed images taken can be clearly displayed.

[0057] The electronic device carried by the client may also be other electronic devices with lower resolution. In this case, due to the limitation of the display resolution of the client, in order to display the monitoring screen of the complete field of view on the client, the high-resolution fused video image is usually required to be reduced to a suitable resolution for display. When the staff needs to track and view the movement and dynamics of a specified target (such as a person, a car, etc.) in the monitoring screen, the local screen where the specified target is located can be selected for enlarged display. For example, by selecting the area where the specified target is located by mouse or touch screen, the client can send the location information of the selected area to the server of the billion-pixel computing imaging system, so that the server can obtain the video data of the local area from the fused video and send it to the client. Since the size of the selected area is small, the client can display the screen containing the detailed information of the specified target that is not reduced or only reduced by a small proportion for the user to view. However, as the specified target moves, the user needs to continuously select a new local screen to enlarge and display, which is less convenient and less efficient. Especially when the specified target moves faster, if the local screen with accurate position cannot be selected in time for enlargement, the continuity of target tracking and monitoring will be affected.

[0058] Based on this background, the applicant has proposed the target tracking and display method of the present application through long-term research and development and experimental verification, which can improve the efficiency of target tracking and monitoring based on billion-pixel fusion video, reduce the difficulty of operation, improve the user experience, and ensure the accuracy and continuity of target tracking display. In addition, it should be noted that the applicant has made a lot of creative efforts in the discovery of the technical problem of the present application and the technical solutions introduced in the following embodiments.

[0059] The target tracking and display method provided in the embodiment of the present application can be applied to Figure 1The billion-pixel computational imaging system shown in the figure. The array camera 102 and the client 106 communicate with the server 104 through the network respectively. The client 106 can be implemented by electronic devices, including but not limited to various personal computers, laptops, smart phones, tablet computers, smart displays, Internet of Things devices, portable wearable devices and other electronic devices with display units, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0060] In one embodiment, Figure 2 As shown, a target tracking display method is provided, which can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0061] Step 202: obtain multiple local videos captured by the array camera, and splice and fuse the local videos to obtain a fused video.

[0062] In implementation, the array camera can be used to shoot video of the target area (such as an airport, park, sea surface, highway, bridge, sports field, etc.). The array camera can send the local video images shot by each sub-camera to the server, so that the server can obtain multiple local video images shot of the target area.

[0063] Then, the server can stitch and fuse a group of local video images taken by each sub-camera at the same time or quasi-same time (because the exposure time of each sub-camera is difficult to be absolutely consistent, there is a certain acceptable time difference in the shooting time of each local video image) to obtain the fused video image corresponding to the moment.

[0064] The stitching and fusion process can adopt the image stitching and fusion method in the related technology. For example, based on the sample local video images taken by the array camera, feature point pairs can be searched based on the overlapping areas between the sample local video images, and then an image stitching and fusion model can be constructed based on the feature point pairs. Then, each group of local video images is stitched and fused through the image stitching and fusion model to obtain a fused video image. In an optional implementation, the array camera can also include a wide-angle camera, and a panoramic image of the target area (covering the entire target area or most of the target area) is captured by the wide-angle camera to find feature point pairs of each sample local video image and the sample panoramic image, and a homography transformation matrix is ​​constructed based on the feature point pairs so that each local video image is converted to the coordinate system of the panoramic image to obtain an image stitching and fusion model.

[0065] It is understandable that the local video image obtained by the server can be an image captured in real time or a pre-recorded video image. In some scenarios, if the field of view covered by all sub-cameras of the array camera is larger than the field of view of the target area to be captured, some sub-cameras can be controlled to capture local video images as needed. Therefore, the server can only obtain the local video images captured by some sub-cameras included in the array camera.

[0066] Step 204, target detection and target tracking are performed on each frame of the fused video image to obtain the fused position information and feature image of each target object in the fused video image, and the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information are correspondingly stored in the target information table.

[0067] Among them, the target object can be a preset type of target such as a person, a car, a drone, an airplane, a bird, etc. The target type of interest can be pre-set, and then the target detection algorithm (such as YOLO, RCNN, Fast-RCNN, etc.) is used to detect the target objects contained in the fused video image to obtain the position information of each target object in the fused video image. The feature image is a local image containing the features of the target object captured from the fused video image or local image in which the target object is detected. The feature image can be a reduced image containing the complete target object, or it can be an image containing only some features of the target object such as a face, a license plate, etc.

[0068] Optionally, step 204 may specifically include the following steps:

[0069] Step 2041 , for a group of local video images taken by each sub-camera at the same time, a target detection algorithm is used to respectively detect the target object contained in each local video image in the group, and obtain the local position information of each target object in the group of local video images.

[0070] Step 2042 , coordinate conversion is performed on the local position information of each target object in the local video image to obtain the fused position information of each target object in the frame fused video image corresponding to the group of local video images.

[0071] Optionally, step 2042 specifically includes the following steps:

[0072] Step 20421, based on the coordinate transformation matrix (such as homography transformation matrix) corresponding to each sub-camera, the local position information of each target object in the local video image is subjected to coordinate transformation processing to obtain the initial fused position information of each target object in the fused video image corresponding to the group of local video images.

[0073] Step 20422, match each target object in each partial video image with each target object in other partial video images in the same group to determine a successfully matched target object, wherein a successfully matched target object refers to the same target object detected in at least two partial video images in the same group.

[0074] Optionally, the matching process includes: calculating the position overlap of any two target objects between any two local video images in the same group based on the initial fusion position information of each target object, and determining the target object whose position overlap is greater than a preset threshold as a successfully matched target object.

[0075] Step 20423, merging multiple initial fused position information of the successfully matched target object to obtain the fused position information of the target object, and determining the initial fused position information of the unmatched target object as the fused position information.

[0076] Since the same target object may be captured by adjacent sub-cameras at the same time, the position overlap of the initial fusion position information is used to determine whether it is the same target, and then the initial fusion position information of the same target is merged (detection frame merging) to improve the detection accuracy of each target object in the fused image.

[0077] Step 2043: Track each target object based on the fusion position information of each target object in the fused video images of adjacent frames, and assign the same global unique identifier to the same target object contained in the fused video images of different frames.

[0078] Step 2044: capture the feature image of each target object in the fused video image based on the fused position information.

[0079] In implementation, for a non-merged target object, a feature image can be captured from a local image, and for a merged target object, a feature image can be captured from a fused image. Optionally, the captured image can be reduced to a preset smaller resolution as a feature image for easy display on the client.

[0080] Step 206, obtaining the current frame fused video image, generating a panoramic playback image based on the current frame fused video image, and sending the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client.

[0081] The panoramic playback image is used by the client to display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and the characteristic image is used by the client to display the characteristic image corresponding to each target object in the target display area of ​​the first display interface. Figure 3The first display interface shown can display a panoramic playback image in the panoramic video playback area 301 and display a feature image corresponding to each target object in the target display area 302 .

[0082] Step 208: receiving a target tracking display request sent by the client, and determining a globally unique identifier of a designated tracking target.

[0083] Optionally, the target tracking display request includes a first type. The user can select a feature image of a specified target object in the target display area 302 to trigger a first type of target tracking display request. The first type of target tracking display request is a target tracking display request generated and sent to the server based on the global unique identifier of the specified target object corresponding to the specified feature image when the client detects the user's selection operation for the specified feature image among the feature images displayed in the target display area. When the server receives the first type of target tracking display request sent by the client, it can obtain the global unique identifier of the specified tracking target carried in the target tracking display request.

[0084] Optionally, step 206 also includes determining the timestamp information of the fused video image of the current frame as the timestamp information of the panoramic playback image, and sending the timestamp information of the panoramic playback image to the client. The target tracking display request includes a second type. The user can select a specified target object in the panoramic video playback area 301 to trigger a second type of target tracking display request. The second type of target tracking display request is that the client detects a target tracking display instruction triggered by the user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information according to the position of the trigger instruction, and generates and sends a target tracking display request to the server based on the specified timestamp information and the specified fusion position information.

[0085] When the server receives the second type of target tracking display request sent by the client, it can obtain the specified timestamp information and the specified fusion position information carried in the target tracking display request, and query the target information table for the globally unique identifier corresponding to the specified timestamp information and the specified fusion position information to obtain the globally unique identifier of the specified tracking target.

[0086] Step 210, based on the globally unique identifier of the designated tracking target and the timestamp information of the new current frame fused video image currently obtained, query the matching target fusion position information in the target information table, obtain the local fusion image corresponding to the target fusion position information in the new current frame fused video image, and send the local fusion image to the client.

[0087] The local fusion image is used by the client to display the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface. Figure 4 The second display interface shown includes a target tracking playback area 401 for tracking and displaying a local fused image including a designated tracking target.

[0088] In the above target tracking display method, the panoramic playback image and the characteristic image of the target object contained in the image can be displayed simultaneously on the current interface by the client. The user can trigger the target tracking display request on the current interface, such as clicking the characteristic image or clicking the location of the target in the panoramic playback image to trigger it. After receiving the target tracking display request, the server queries the position information of the specified tracking target in the current frame fusion video image (usually a frame of fusion video image obtained by the current latest fusion) in the target information table according to the global unique identifier of the specified tracking target for the current frame fusion video image at each subsequent moment, and then intercepts the local fusion image of the corresponding position in the current frame fusion video image and sends it to the client for display. Since the local fusion image mainly contains the pixel points of the specified tracking target, the size is much smaller than the size of the original fusion video image (usually smaller than the size of a frame of local video image), and can match the display resolution of the client without compression, the local fusion image displayed by the client can retain the detailed information of the specified tracking target to meet the needs of the staff to track and view the dynamics of the target. When watching the panoramic playback video, the staff can trigger the tracking display of any target in the current playback screen at any time as needed, which is convenient to operate, efficient in target tracking display, and can ensure the accuracy and continuity of target tracking display.

[0089] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0090] Based on the same inventive concept, the embodiment of the present application also provides a target tracking display device for implementing the target tracking display method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more target tracking display device embodiments provided below can refer to the limitations of the target tracking display method above, and will not be repeated here.

[0091] In one embodiment, Figure 5 As shown, a target tracking display device 500 for a billion-pixel computational imaging system is provided, comprising: a fusion module 501, a target tracking module 502, a first sending module 503, a determination module 504, and a second sending module 505, wherein:

[0092] The fusion module 501 is used to obtain multiple local videos captured by the array camera, and to splice and fuse the local videos to obtain a fused video.

[0093] The target tracking module 502 is used to perform target detection and target tracking on each frame of the fused video image, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table.

[0094] The first sending module 503 is used to obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in the target display area of ​​the first display interface.

[0095] The determination module 504 is configured to receive the target tracking display request sent by the client and determine the globally unique identifier of the specified tracking target.

[0096] The second sending module 505 is used to query the matching target fusion position information in the target information table based on the global unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently obtained, and obtain the local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client, so that the client can display the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface.

[0097] In one embodiment, the target tracking module 502 is specifically used to: for a group of local video images taken by each of the sub-cameras at the same time, use a target detection algorithm to respectively detect the target object contained in each of the local video images in the group, and obtain the local position information of each target object in the group of local video images; perform coordinate transformation processing on the local position information of each target object in the local video image to obtain the fusion position information of each target object in the frame fusion video image corresponding to the group of local video images; based on the fusion position information of each target object in the adjacent frame fusion video image, track each target object, and assign the same global unique identifier to the same target object contained in different frame fusion video images; and capture the feature image of each target object in the fusion video image based on the fusion position information.

[0098] In one embodiment, the target tracking module 502 is specifically used to: based on the coordinate transformation matrix corresponding to each of the sub-cameras, perform coordinate transformation processing on the local position information of each target object in the local video image to obtain the initial fusion position information of each target object in the fusion video image corresponding to the group of local video images; match each target object in each of the local video images with each target object in other local video images in the same group to determine the successfully matched target object; the successfully matched target object refers to the same target object detected in at least two local video images in the same group; merge multiple initial fusion position information of the successfully matched target object to obtain the fusion position information of the target object, and determine the initial fusion position information of the unmatched target object as the fusion position information.

[0099] In one of the embodiments, the target tracking module 502 is specifically used to: calculate the position overlap of any two target objects between any two local video images in the same group based on the initial fusion position information of each target object, and determine the target object whose position overlap is greater than a preset threshold as a successfully matched target object.

[0100] In one of the embodiments, the determination module 504 is specifically used to: upon receiving a first type of target tracking display request sent by the client, obtain a globally unique identifier of a specified tracking target carried by the target tracking display request; the first type of target tracking display request is a target tracking display request generated and sent to the server by the client when it detects a user's selection operation on a specified feature image among the feature images displayed in the target display area, based on the globally unique identifier of a specified target object corresponding to the specified feature image.

[0101] In one of the embodiments, the device further includes a third sending module, which is used to determine the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and send the timestamp information of the panoramic playback image to the client. The determination module 504 is specifically used to: when receiving the second type of target tracking display request sent by the client, obtain the specified timestamp information and the specified fusion position information carried by the target tracking display request; the second type of target tracking display request is that the client detects the target tracking display instruction triggered by the user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information according to the position of the trigger instruction, generates and sends the target tracking display request to the server based on the specified timestamp information and the specified fusion position information; in the target information table, query the global unique identifier corresponding to the specified timestamp information and the specified fusion position information to obtain the global unique identifier of the specified tracking target.

[0102] Each module in the above target tracking and display device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0103] The embodiment of the present application also provides a billion-pixel computational imaging system, including an array camera, a server, and a client, wherein the array camera includes a plurality of sub-cameras, wherein:

[0104] The array camera is used to shoot a video of a target scene through multiple sub-cameras, and send the shot multiple local videos to a server;

[0105] The server is used for splicing and fusing the acquired partial videos to obtain a fused video;

[0106] The server is also used to perform target detection and target tracking on each frame of the fused video image of the fused video, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table; obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the global unique identifier and feature image of each target object in the current frame fused video image to the client;

[0107] The client is used to receive the panoramic playback image sent by the server and the globally unique identifier and feature image of each target object in the fused video image corresponding to the panoramic playback image, display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and display the feature image corresponding to each target object in the target display area of ​​the first display interface;

[0108] The client is also used to send a target tracking display request to the server in response to a user's trigger operation on a designated tracking object;

[0109] The server is further configured to receive a target tracking display request sent by the client, determine a globally unique identifier of a specified tracking target, query matching target fusion position information in the target information table based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently acquired, intercept a local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client;

[0110] The client is further configured to display the local fusion image including the designated tracking target in the target tracking playback area of ​​the second display interface.

[0111] In one of the embodiments, the server is further used to: for a group of local video images taken by each of the sub-cameras at the same time, use a target detection algorithm to respectively detect the target object contained in each of the local video images in the group, and obtain the local position information of each target object in the group of local video images; perform coordinate transformation processing on the local position information of each target object in the local video image, and obtain the fusion position information of each target object in the frame fusion video image corresponding to the group of local video images; based on the fusion position information of each target object in the adjacent frame fusion video images, track each target object, and assign the same global unique identifier to the same target object contained in different frame fusion video images; based on the fusion position information, capture the feature image of each target object in the fusion video image.

[0112] In one of the embodiments, the server is further used to: perform coordinate transformation processing on the local position information of each target object in the local video image based on the coordinate transformation matrix corresponding to each sub-camera to obtain the initial fusion position information of each target object in the fusion video image corresponding to the group of local video images; match each target object in each of the local video images with each target object in other local video images in the same group to determine the successfully matched target object; the successfully matched target object refers to the same target object detected in at least two local video images in the same group; merge multiple initial fusion position information of the successfully matched target object to obtain the fusion position information of the target object, and determine the initial fusion position information of the unmatched target object as the fusion position information.

[0113] In one of the embodiments, the server is further used to: calculate the position overlap of any two target objects between any two local video images in the same group based on the initial fusion position information of each target object, and determine the target object whose position overlap is greater than a preset threshold as a successfully matched target object.

[0114] In one of the embodiments, the receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes:

[0115] The client is also used to detect the user's selection operation on a specified feature image among the feature images displayed in the target display area, determine the globally unique identifier of the specified target object corresponding to the specified feature image, and send a first type target tracking display request carrying the globally unique identifier of the specified target object to the server.

[0116] The server is further configured to receive a first type target tracking display request sent by the client, and obtain a globally unique identifier of a designated tracking target carried in the target tracking display request.

[0117] In one of the embodiments, the server is further configured to determine the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and send the timestamp information of the panoramic playback image to the client.

[0118] The client is also used to detect a target tracking display instruction triggered by a user at any position in the panoramic video playback area, determine the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determine the specified fusion position information according to the position of the triggering instruction, and send a second type of target tracking display request containing the specified timestamp information and the specified fusion position information to the server.

[0119] The server is also used to receive a second type of target tracking display request sent by the client, obtain the specified timestamp information and the specified fusion position information carried by the target tracking display request; query the target information table for a globally unique identifier corresponding to the specified timestamp information and the specified fusion position information, and obtain the globally unique identifier of the specified tracking target.

[0120] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required or generated for executing the above-mentioned target tracking and display method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target tracking and display method is implemented.

[0121] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0123] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0124] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0127] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A target tracking and display method for a billion-pixel computational imaging system, characterized in that: The method comprises: Acquire multiple local videos captured by the array camera, and splice and fuse the local videos to obtain a fused video; Performing target detection and target tracking on each frame of the fused video image of the fused video to obtain fused position information and feature images of each target object in the fused video image, and correspondingly storing the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table; Acquire a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to the client; the panoramic playback image is used by the client to display the panoramic playback image in a panoramic video playback area of ​​a first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in a target display area of ​​the first display interface; Receiving a target tracking display request sent by the client, and determining a globally unique identifier of a designated tracking target; Based on the globally unique identifier of the designated tracking target and the timestamp information of the new current frame fused video image currently obtained, the matching target fusion position information is queried in the target information table, and a local fusion image corresponding to the target fusion position information is obtained in the new current frame fusion video image, and the local fusion image is sent to the client, so that the client displays the local fusion image containing the designated tracking target in the target tracking playback area of ​​the second display interface.

2. The method according to claim 1, characterized in that The performing target detection and target tracking on each frame of the fused video image to obtain fused position information and feature images of each target object in the fused video image includes: For a group of local video images taken by each sub-camera included in the array camera at the same time, a target detection algorithm is used to respectively detect the target object included in each local video image in the group, and obtain the local position information of each target object in the group of local video images; Performing coordinate transformation on the local position information of each target object in the local video image to obtain fused position information of each target object in the frame fused video image corresponding to the group of local video images; Based on the fusion position information of each target object in the fused video images of adjacent frames, each target object is tracked, and a same global unique identifier is assigned to the same target object contained in the fused video images of different frames; A feature image of each of the target objects is obtained in the fused video image based on the fused position information.

3. The method according to claim 2, characterized in that The coordinate transformation of the local position information of each target object in the local video image to obtain the fused position information of each target object in the frame fused video image corresponding to the group of local video images includes: Based on the coordinate transformation matrix corresponding to each of the sub-cameras, coordinate transformation is performed on the local position information of each target object in the local video image to obtain the initial fused position information of each target object in the fused video image corresponding to the group of local video images; Matching each target object in each of the local video images with each target object in other local video images in the same group to determine a successfully matched target object; the successfully matched target object refers to the same target object detected in at least two local video images in the same group; The multiple initial fused position information of the successfully matched target object are merged to obtain the fused position information of the target object, and the initial fused position information of the unsuccessfully matched target object is determined as the fused position information.

4. The method according to claim 3, characterized in that: The matching process of each target object in each of the partial video images with each target object in other partial video images in the same group to determine the successfully matched target object includes: Based on the initial fusion position information of each target object, the position overlap of any two target objects between any two local video images in the same group is calculated, and the target object whose position overlap is greater than a preset threshold is determined as a successfully matched target object.

5. The method according to claim 1, characterized in that The receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes: Upon receiving a first type of target tracking display request sent by the client, a globally unique identifier of a designated tracking target carried in the target tracking display request is obtained; the first type of target tracking display request is a target tracking display request generated and sent to a server based on the globally unique identifier of a designated tracking target corresponding to the designated feature image, when the client detects a user's selection operation on a designated feature image among the feature images displayed in the target display area.

6. The method according to claim 1, characterized in that After the panoramic playback image is generated based on the current frame fusion video image, the method further includes: Determine the timestamp information of the current frame fused video image as the timestamp information of the panoramic playback image, and send the timestamp information of the panoramic playback image to the client; The receiving the target tracking display request sent by the client and determining the globally unique identifier of the specified tracking target includes: In the case of receiving the second type of target tracking display request sent by the client, obtaining the specified timestamp information and the specified fusion position information carried by the target tracking display request; the second type of target tracking display request is that the client detects a target tracking display instruction triggered by a user at any position in the panoramic video playback area, determines the timestamp information of the panoramic playback image displayed when the instruction is triggered as the specified timestamp information, determines the specified fusion position information according to the position of the triggering instruction, and generates and sends the target tracking display request to the server based on the specified timestamp information and the specified fusion position information; The global unique identifier corresponding to the specified timestamp information and the specified fusion position information is searched in the target information table to obtain the global unique identifier of the specified tracking target.

7. A billion-pixel computational imaging system, characterized in that: The billion-pixel computational imaging system comprises an array camera, a server and a client, wherein the array camera comprises a plurality of sub-cameras, wherein: The array camera is used to shoot a video of a target scene through multiple sub-cameras, and send the shot multiple local videos to a server; The server is used for splicing and fusing the acquired partial videos to obtain a fused video; The server is also used to perform target detection and target tracking on each frame of the fused video image of the fused video, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table; obtain the current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the global unique identifier and feature image of each target object in the current frame fused video image to the client; The client is used to receive the panoramic playback image sent by the server and the globally unique identifier and feature image of each target object in the fused video image corresponding to the panoramic playback image, display the panoramic playback image in the panoramic video playback area of ​​the first display interface, and display the feature image corresponding to each target object in the target display area of ​​the first display interface; The client is also used to send a target tracking display request to the server in response to a user's trigger operation on a designated tracking object; The server is further configured to receive a target tracking display request sent by the client, determine a globally unique identifier of a specified tracking target, query matching target fusion position information in the target information table based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently acquired, intercept a local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client; The client is further configured to display the local fusion image including the designated tracking target in the target tracking playback area of ​​the second display interface.

8. A target tracking display device, characterized in that: The device comprises: A fusion module is used to obtain multiple local videos captured by the array camera, and to splice and fuse the local videos to obtain a fused video; A target tracking module is used to perform target detection and target tracking on each frame of the fused video image, obtain the fused position information and feature image of each target object in the fused video image, and store the global unique identifier of the target object, the timestamp information of the fused video image and the fused position information in a target information table; A first sending module is used to obtain a current frame fused video image, generate a panoramic playback image based on the current frame fused video image, and send the panoramic playback image and the globally unique identifier and feature image of each target object in the current frame fused video image to a client; the panoramic playback image is used by the client to display the panoramic playback image in a panoramic video playback area of ​​a first display interface, and the feature image is used by the client to display the feature image corresponding to each target object in a target display area of ​​the first display interface; A determination module, configured to receive a target tracking display request sent by the client and determine a globally unique identifier of a specified tracking target; The second sending module is used to query the matching target fusion position information in the target information table based on the globally unique identifier of the specified tracking target and the timestamp information of the new current frame fusion video image currently acquired, and to intercept the local fusion image corresponding to the target fusion position information in the new current frame fusion video image, and send the local fusion image to the client, so that the client can display the local fusion image containing the specified tracking target in the target tracking playback area of ​​the second display interface.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images