A method and apparatus for assisting epidemiological investigation based on machine vision and deep learning

Abstract

This invention discloses a method and device for assisting epidemiological investigation based on machine vision and deep learning. The method includes acquiring the target to be tracked, single-lens target tracking, mask detection and classification and storage of at-risk individuals, cross-lens target re-identification, and target trajectory mapping. It innovatively applies pedestrian re-identification technology to epidemiological investigation and tracing, automatically generating trajectories and close contacts, significantly reducing manual labor while improving the accuracy and timeliness of the investigation. Furthermore, this system has no hardware requirements and can be directly integrated into existing monitoring systems, allowing for remote maintenance and easy upgrades. Overall, this software outperforms existing epidemiological investigation methods and has advantages such as strong versatility, wide applicability, and ease of use.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and particularly relates to a method and apparatus for assisting epidemiological investigation based on machine vision and deep learning. Background Technology

[0002] Currently, some methods are available for completing contact tracing, such as:

[0003] Option 1: Inquire about the travel history of infected individuals through manual questioning, and use information such as public transportation to identify potential close contacts in areas near the infected individual's travel history, and then conduct further manual questioning for confirmation; Option 2: Deploy venue QR codes on a large scale, requiring pedestrians to scan the venue QR codes upon entering the venue, and when an infected individual is found, directly use big data analysis to identify potential close contacts who have scanned the venue QR codes of the locations the infected individual has visited.

[0004] The following problems exist in the two mainstream solutions mentioned above:

[0005] (1) The first approach is subject to interference from factors such as the infected person’s forgetfulness, and the obtained trajectory of the infected person is not accurate enough. It also has the problems of being time-consuming and requiring manpower inefficiently.

[0006] (2) The second option also has the problem of pedestrians not cooperating and not scanning the venue code, which makes the final result difficult to be accurate.

[0007] Both of the above methods can obtain the travel history of infected persons to a certain extent, but they have problems such as low accuracy and low efficiency. Summary of the Invention

[0008] In view of the shortcomings of the prior art, the purpose of this application is to provide a method and apparatus for assisting epidemiological investigation based on machine vision and deep learning.

[0009] According to a first aspect of the embodiments of this application, a method for assisting epidemiological investigation based on machine vision and deep learning is provided, comprising:

[0010] (1) Acquisition of target to be tracked: Acquire video and video frames in which the target is not occluded in the video, perform pedestrian detection on the video frames to obtain several selection boxes, select the corresponding selection box as the selection box of the target to be tracked according to the received user instruction, and save the image of the target to be tracked in the corresponding selection box and the corresponding time.

[0011] (2) Single-lens target tracking: Based on the target image to be tracked and the corresponding time saved in step (1), the target to be tracked is tracked forward and backward respectively to obtain all video frames in which the target to be tracked appears in the video, record the trajectory information of the target to be tracked and save each video frame and the image inside the rectangular frame of the target to be tracked, the rectangular frame coordinates and the time.

[0012] (3) Mask detection and classification and storage of at-risk personnel: all video frames in which the target to be tracked appears in the video are subjected to pedestrian detection and mask detection, and then classified and stored by a re-identification algorithm;

[0013] (4) Cross-camera target re-identification: After the target to be tracked disappears from the video, the target to be tracked is searched in the video of the corresponding camera in the map adjacency matrix representing the geographical location of the camera to obtain the complete trajectory information of the target to be tracked.

[0014] (5) Target trajectory drawing: Based on the complete trajectory information of the target to be tracked, draw the trajectory of the target to be tracked on the map.

[0015] Further, step (1) includes the following sub-steps:

[0016] (1.1) After obtaining a video frame in which the target to be tracked is not occluded, the first target detection algorithm is used to perform pedestrian recognition on the video frame, and the recognition result is marked with a box in the video frame to generate several selection boxes for the user to select the target to be tracked.

[0017] (1.2) Select the corresponding selection box from the received user instruction as the selection box for the target to be tracked, and save the image of the target to be tracked in the corresponding selection box and the corresponding time.

[0018] Furthermore, step (2) includes the following sub-steps:

[0019] (2.1) Obtain T from the video in the database. s The images at each time point are used for pedestrian recognition, resulting in k bounding boxes R. Ts1 R Ts2 …R Tsk (k>=1), perform overlap matching on all the bounding boxes in the image with the selection box of the target to be tracked, and obtain the bounding box R corresponding to the target to be tracked. w1 And record the relevant coordinate information;

[0020] (2.2) From T s Start loading video frames forward and perform pedestrian detection on each frame using the first object detection algorithm, with a time interval of T.n , thus obtaining the i-shaped rectangle R Tn1 R Tn2 …R Tni (i>=1), and simultaneously, a multi-target tracking algorithm is used to track the target until the bounding box of the target disappears in the video. The forward trajectory information and the disappearance time T of the target in the video are recorded. f Save each video frame along with the corresponding image within the bounding box of the target to be tracked, the bounding box coordinates, and the time.

[0021] (2.3) From T s Start loading video frames sequentially and perform pedestrian detection on each frame using the first object detection algorithm, with a time interval of T. m Frame, obtain j rectangular boxes R Tm1 R Tm2 …R Tmj (j>=1), and simultaneously, a multi-target tracking algorithm is used to track the target until the bounding box of the target disappears from the video. The backward trajectory information and the disappearance time T of the target within the video are recorded. l Each video frame, along with the corresponding image within the bounding box of the target to be tracked, the bounding box coordinates, and the time, is saved.

[0022] Furthermore, step (2) also includes the following sub-steps:

[0023] (2.4) For the images within the rectangular bounding box of the target to be tracked saved in steps (2.2) and (2.3), the deletion time is less than T. f +1 or greater than T l -1 images are used to remove incomplete images that may appear and disappear when the target to be tracked is detected.

[0024] Furthermore, step (3) includes the following sub-steps:

[0025] (3.1) In all video frames in which the target to be tracked appears, the second target detection algorithm is used to detect whether the corresponding pedestrians are wearing masks in the images within the rectangular frames other than the target to be tracked, and then risk classification is performed.

[0026] (3.2) Input the images inside the rectangles other than the target rectangle into the re-identification network to determine whether the corresponding pedestrian has been identified. If the corresponding pedestrian has been identified, save the images inside the rectangles, the corresponding pedestrian ID, time, and rectangle coordinates to the corresponding dataset. Otherwise, create a new dataset and save the images inside the rectangles, the corresponding pedestrian ID, time, and rectangle coordinates to the new dataset.

[0027] Furthermore, step (4) includes the following sub-steps:

[0028] (4.1) Based on the map adjacency matrix, directly obtain x cameras (x>=1) that are geographically adjacent to the camera corresponding to the video, and load these x cameras C1, C2…C…in sequence. k From (T) f +T a ) to (T f +T a +ΔT) video clip, where T a The simplified search duration is obtained based on the map adjacency matrix, where ΔT is the search duration. Each video segment and its corresponding start and end times are saved.

[0029] (4.2) Perform pedestrian detection using the first target detection algorithm on the video frames in each video segment, resize the image within the obtained rectangle to match the input size of the re-identification network in step (3), adjust the average brightness of the image to the same average brightness level as the video frame in step (1), and save the image and the corresponding time and coordinates.

[0030] (4.3) Input the image saved in step (4.2) and the image within the rectangular frame of the target to be tracked saved in step (2) into the re-identification network to determine whether the target to be tracked exists in the image;

[0031] (4.4) If there is, return to step (2); if not, expand the search time range and return to step (4.1) to search again. If the search continues to fail, continuously expand the search time range until the search is successful or all video segments are obtained, thereby obtaining the complete trajectory information of the target to be tracked, where ΔT1 is the search duration.

[0032] Furthermore, step (5) includes the following sub-steps:

[0033] (5.1) Read the complete trajectory information of the target to be tracked in sequence to obtain the two cameras C1 and C2 that pass by in sequence;

[0034] (5.2) Use the already marked map to obtain the coordinate positions P1 and P2 of the two cameras on the map, and perform binarization on the map to obtain road information;

[0035] (5.3) Use the growth algorithm to obtain the most reasonable route between points P1 and P2 in the binarized map, record the coordinates of each point in the route, and then draw the route on the original map.

[0036] (5.4) Repeat steps (5.1) to (5.3) until all the recorded trajectory information of the target to be tracked is read.

[0037] According to a second aspect of the embodiments of this application, a contact tracing assistance device based on machine vision and deep learning is provided, comprising:

[0038] Target acquisition module: acquires video and video frames in which the target is not occluded, performs pedestrian detection on the video frames to obtain several selection boxes, selects the corresponding selection box as the selection box for the target to be tracked based on the received user instruction, and saves the image of the target to be tracked in the corresponding selection box and the corresponding time.

[0039] Single-lens target tracking module: Based on the target image and corresponding time stored in the target acquisition module, the target is tracked forward and backward respectively to obtain all video frames in which the target appears in the video, record the trajectory information of the target and save the image of the target within the rectangle, the rectangle coordinates and time corresponding to each video frame;

[0040] Mask detection and at-risk personnel classification and storage module: Pedestrian detection and mask detection are performed on all video frames in which the target to be tracked appears in the video, and then the classification and storage are performed through a re-identification algorithm;

[0041] Cross-camera target re-identification module: After the target to be tracked disappears from the video, the module searches the video of the corresponding camera in the map adjacency matrix, which represents the geographical location of the camera, to see if the target to be tracked exists, thereby obtaining the complete trajectory information of the target to be tracked.

[0042] Target trajectory drawing module: Based on the complete trajectory information of the target to be tracked, draw the trajectory of the target to be tracked on the map.

[0043] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0044] One or more processors;

[0045] Memory, used to store one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0047] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0048] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0049] As demonstrated by the above embodiments, this application innovatively applies pedestrian re-identification technology to epidemiological investigation and tracing, automatically generating trajectories to be tracked and close contacts after risk classification. This significantly reduces manual labor while improving the accuracy and duration of epidemiological investigations. Furthermore, this system has no hardware requirements and can be directly integrated into existing monitoring systems for remote maintenance and easy upgrades. Overall, this software outperforms existing epidemiological investigation methods and offers advantages such as strong versatility, wide applicability, and ease of use.

[0050] It should be noted that the parties involved have acknowledged and consented to all identification and tracking involved in this application.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] Figure 1 This is a flowchart illustrating an epidemiological investigation assistance method based on machine vision and deep learning, according to an exemplary embodiment.

[0054] Figure 2 It is a screenshot of a video clip of a target to be tracked, according to an exemplary embodiment.

[0055] Figure 3 This is an example of a target to be tracked on a travel trajectory.

[0056] Figure 4 This is a block diagram illustrating an epidemiological investigation assistance device based on machine vision and deep learning, according to an exemplary embodiment. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0058] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0059] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0060] Figure 1 This is a flowchart illustrating an epidemiological investigation assistance method based on machine vision and deep learning, according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps:

[0061] (1) Acquisition of target to be tracked: Acquire video and video frames in which the target is not occluded in the video, perform pedestrian detection on the video frames to obtain several selection boxes, select the corresponding selection box as the selection box of the target to be tracked according to the received user instruction, and save the image of the target to be tracked in the corresponding selection box and the corresponding time.

[0062] (2) Single-lens target tracking: Based on the target image to be tracked and the corresponding time saved in step (1), the target to be tracked is tracked forward and backward respectively to obtain all video frames in which the target to be tracked appears in the video, record the trajectory information of the target to be tracked and save the image inside the rectangle of the target to be tracked, the rectangle coordinates and the time corresponding to each video frame.

[0063] (3) Mask detection and classification and storage of at-risk personnel: all video frames in which the target to be tracked appears in the video are subjected to pedestrian detection and mask detection, and then classified and stored by a re-identification algorithm;

[0064] (4) Cross-camera target re-identification: After the target to be tracked disappears from the video, the target to be tracked is searched in the video of the corresponding camera in the map adjacency matrix representing the geographical location of the camera to obtain the complete trajectory information of the target to be tracked.

[0065] (5) Target trajectory drawing: Based on the complete trajectory information of the target to be tracked, draw the trajectory of the target to be tracked on the map.

[0066] As can be seen from the above embodiments, this application innovatively applies pedestrian re-identification technology to epidemiological investigation and source tracing, automatically generating trajectories and close contacts, greatly reducing manual labor while improving the accuracy and shortening the investigation time. Furthermore, this system has no hardware requirements and can be directly integrated into existing monitoring systems, allowing for remote maintenance and easy upgrades. Overall, this software outperforms existing epidemiological investigation methods and has advantages such as strong versatility, wide applicability, and ease of use.

[0067] The specific implementation of step (1) may include the following sub-steps:

[0068] (1.1) After obtaining a video frame in which the target to be tracked is not occluded, the first target detection algorithm is used to perform pedestrian recognition on the video frame, and the recognition result is marked with a box in the video frame to generate several selection boxes for the user to select the target to be tracked.

[0069] Specifically, the user retrieves all surveillance camera videos from the time periods and road segments that the target to be tracked may have passed through, sorts them appropriately, and saves them to a designated folder. The user selects the corresponding video in the system, and the system automatically plays the selected video. The user views the video, and when the target to be tracked appears in the video frame clearly and without obstruction, the user takes a screenshot. The system then obtains the image of the current frame of the video and uses the corresponding camera number and time T to determine the image. s Save the files to the appropriate folder with filenames, and use the pre-trained Yolov5 network to perform pedestrian recognition on the current frame of the obtained video, obtaining k bounding boxes R. Ts1 R Ts2 …R Tsk (k>=1), and draw the rectangle on the original image, and then generate and display an image with several clickable selection boxes on the system interface using the Qt library.

[0070] It should be noted that the first target detection algorithm in this embodiment is YOLOv5. In specific implementations, methods such as R-CNN, Fast R-CNN, Faster R-CNN, SSD, and YOLO can be used. This setting is a conventional setting in this field and will not be elaborated here.

[0071] (1.2) Select the corresponding selection box from the received user instruction as the selection box for the target to be tracked, and save the image of the target to be tracked in the corresponding selection box and the corresponding time;

[0072] Specifically, the user selects the rectangle R corresponding to the target to be tracked by clicking with the mouse.w Save the rectangle R w Images are added to a multi-query dataset, named according to the rule "video name_current time". A record database is also created, formatted as line-by-line text, with each line containing information in the format 'video name, current time, x1 of top-left corner of rectangle, y1 of top-left corner of rectangle, x2 of bottom-right corner of rectangle, y2 of bottom-right corner of rectangle'. It should be noted that the rectangle R... w The naming rules for images are not unique; as long as they reflect the video from which the image originates and the corresponding time, that's sufficient. Similarly, the text format within a record database is not unique either; as long as it reflects the corresponding information, that's sufficient.

[0073] The specific implementation of step (2) may include the following sub-steps:

[0074] (2.1) Obtain T from the video in the database. s The images at each time point are used for pedestrian recognition, resulting in k bounding boxes R. Ts1 R Ts2 …R Tsk (k>=1), perform overlap matching on all the bounding boxes in the image with the selection box of the target to be tracked, and obtain the bounding box R corresponding to the target to be tracked. w1 And record the relevant coordinate information;

[0075] Specifically, the same Yolov5 network is used to record time T. s Pedestrian recognition is performed on video frames to obtain k bounding boxes R. Ts1 R Ts2 …R Tsk (k>=1), then save the rectangle R in step (1) w The image is then matched with the k bounding boxes obtained in this step to determine their overlap, resulting in a specific bounding box R corresponding to the target person to be tracked. w1 Record the relevant coordinate information. The formula for calculating the overlap is as follows:

[0076]

[0077] Where n∈[1,k],R w ∩R Tsn For two rectangular frames R w ,R Tsn The overlapping part, S(R) w ) represents a rectangular frame R w area min(S(R) w ),S(R Tsn R represents the minimum area of ​​the two rectangles. TsnThe rectangle with the highest overlap is R. w1 .

[0078] (2.2) From T s Start loading video frames forward and perform pedestrian detection on each frame using the first object detection algorithm, with a time interval of T. n , thus obtaining the i-shaped rectangle R Tn1 R Tn2 …R Tni (i>=1), and simultaneously, a multi-target tracking algorithm is used to track the target until the bounding box of the target disappears in the video. The forward trajectory information and the disappearance time T of the target in the video are recorded. f Save each video frame along with the corresponding image within the bounding box of the target to be tracked, the bounding box coordinates, and the time.

[0079] Specifically, from T s Start loading video frames forward and performing pedestrian detection using a Yolov5 network, with a time interval of T. n The frame obtains k rectangular boxes R Tn1 R Tn2 …R Tnk (k>=1), and at the same time, the DeepSort algorithm uses motion information to map the bounding boxes of pedestrians detected in each frame to the same pedestrian (i.e., tracking). The principle of this algorithm is as follows:

[0080] The object model is described, representing and used to propagate the target's identity to the next frame. Our approximate inter-frame displacements all have a linear isovelocity model independent of other objects and camera motion. The model for each target's state is as follows:

[0081]

[0082] Where u and v represent the center coordinates of the target, r and h represent the height and aspect ratio of the target's bounding box (BBox), respectively. Note that the aspect ratio should be constant. The last four values ​​represent the predicted next frame. A standard Kalman filter with a constant velocity motion and linear observation model is used to treat the above 8-dimensional state as a direct observation model of the object's state. For each trajectory, there is a threshold 'a' to record the time from the last successful match to the current moment. When this value exceeds the pre-set threshold 'A'... max If a trajectory fails to match for an extended period, it is considered terminated; intuitively, a trajectory that cannot be matched for a long time is considered to have ended. Then, during the matching process, any detection that fails to match is considered as potentially generating a new trajectory.

[0083] In Deepsort, Mahalanobis distance is used to evaluate the predicted Kalman state and the new state:

[0084]

[0085] S represents the motion matching degree between the j-th detection and the i-th trajectory, where S i It is the covariance matrix of the trajectory in the observation space at the current moment, obtained by the Kalman filter prediction. i It is the predicted observation of the trajectory at the current moment, d j Let (u, v, r, h) be the state of the j-th detection. Considering the continuity of motion, the detections can be filtered using this Mahalanobis distance, and a threshold t can be set. (1) As a threshold, we can define a threshold function.

[0086]

[0087] Mahalanobis distance is a good correlation metric when the target motion uncertainty is low. However, in practice, camera movement can cause a large number of Mahalanobis distance mismatches, rendering this metric ineffective. Therefore, a second metric needs to be integrated for each bounding box (BBox). j Calculate a surface feature descriptor r j ,r j =1, then create another gallery to store the latest L k A descriptor for a trajectory, namely Then, the minimum cosine distance between the i-th trajectory and the j-th trajectory is used as the second metric:

[0088]

[0089] We can also represent it using a threshold function:

[0090]

[0091] Next, we merge these two scales into:

[0092] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0093]

[0094] In summary, distance metrics are effective for short-term prediction and matching, while appearance information is more effective for measuring matching degree for trajectories lost over a long period of time.

[0095] Record the disappearance time T until the bounding box of the target object disappears from the camera's video feed. fThis refers to the time when the target to be tracked appears in the current video segment.

[0096] Create an incremental directory to save images of the target to be tracked within the bounding boxes of each video clip. The naming convention is "Video Name_Current Time_Top Left Corner X-coordinate x1_Top Left Corner Y-coordinate y1_Bottom Right Corner X-coordinate x2_Bottom Right Corner Y-coordinate y2". It should be noted that this naming convention is not unique; it only needs to reflect the bounding box coordinates and the current time.

[0097] It should be noted that the multi-target tracking algorithm in this embodiment is DeepSORT, but in specific implementations, SORT, JDE algorithm, FAIRMOT algorithm, etc. can also be used. This setting is a conventional setting in this field and will not be elaborated here.

[0098] (2.3) From T s Start loading video frames sequentially and perform pedestrian detection on each frame using the first object detection algorithm, with a time interval of T. m Frame, obtain j rectangular boxes R Tm1 R Tm2 …R Tmj (j>=1), and simultaneously, a multi-target tracking algorithm is used to track the target until the bounding box of the target disappears from the video. The backward trajectory information and the disappearance time T of the target within the video are recorded. l Save each video frame along with the corresponding image within the bounding box of the target to be tracked, the bounding box coordinates, and the time.

[0099] Specifically, steps (2.3) are the same as steps (2.2), and will not be repeated here.

[0100] Preferably, step (2) may further include sub-step (2.4):

[0101] (2.4) For the images within the rectangular bounding box of the target to be tracked saved in steps (2.2) and (2.3), the deletion time is less than T. f +1 or greater than T l -1 images are used to remove incomplete images that may appear and disappear when the target to be tracked appears;

[0102] Specifically, the time when the image was acquired is obtained by reading the name of the image file, and the deletion time is less than T. f +1 image and time greater than T l -1 images are used to remove incomplete images that may appear and disappear when pedestrians are present.

[0103] The specific implementation of step (3) may include the following sub-steps:

[0104] (3.1) In all video frames in which the target to be tracked appears, the second target detection algorithm is used to detect whether the corresponding pedestrians are wearing masks in the images within the rectangular frames other than the target to be tracked, and then risk classification is performed.

[0105] Specifically, the bounding boxes except for the target bounding box in step (2) are input into the SSD network. First, the image size is modified to the SSD network input size of 360*360 pixels. Several possible bounding boxes of faces wearing masks or without masks are identified. Then, the non-maximum suppression algorithm (NMS) is used to remove invalid bounding boxes based on the bounding box overlap (calculated using the above overlap calculation formula) and the SSD network classification score. This yields the result of whether the pedestrian in the input image is wearing a mask. Based on this, bounding boxes with different labels are drawn on the video frame, i.e., risk classification. The bounding box labeled '0' contains the target to be tracked, the bounding box labeled 'nomask' contains pedestrians without masks, and the bounding box labeled 'mask' contains pedestrians wearing masks.

[0106] It should be noted that the SSD algorithm is used as the second object detection algorithm in this embodiment. In specific implementations, R-CNN, Fast R-CNN, Faster R-CNN, SSD, YOLO, etc. can also be used. This setting is a conventional setting in the field and will not be elaborated here.

[0107] (3.2) Input the images inside the rectangles except the target rectangle to be tracked into the re-identification network to determine whether the corresponding pedestrian has been identified. If the corresponding pedestrian has been identified, save the images inside the rectangles, the corresponding pedestrian number, time, and rectangle coordinates to the corresponding dataset. Otherwise, create a new dataset and save the images inside the rectangles, the corresponding pedestrian number, time, and rectangle coordinates to the new dataset.

[0108] Specifically, a re-identification network is used to classify and save pedestrians who are in close contact with the target to be tracked: an incremental catalog is created to save pedestrians who are in close contact with the target. If this step is being performed for the first time, the images within the bounding boxes (excluding the target) are saved sequentially in a newly created subset of the contact dataset. If this step is not being performed for the first time, the images within the bounding boxes (excluding the target) and all images in the contact dataset are input into a ResNet50-based re-identification network. The principle of this network is as follows:

[0109] Given a 64*128 RGB image as input, an average pooling layer is added to the last layer of the ResNet50 network to pool the 2048*7*7 dimensional output into a 2048*1*1 dimensional vector. This 2048 dimensional vector is then fed into a fully connected layer to become a 512 dimensional vector as the network output. For different output images, the Euclidean distance between the corresponding 512 dimensional vectors of two images is calculated to determine if they belong to the same person. The Euclidean distance is calculated as follows:

[0110]

[0111] If an image within a bounding box belongs to the same pedestrian as an image in the `contact` dataset (e.g., an image within the nth subset of the `contact` dataset), then this image is stored in the nth subset of the `contact` dataset. Otherwise, a new subset is created within the `contact` dataset to store this image. When saving images in this step, the naming convention is "Pedestrian ID_Video Name_Current Time_Top-left corner x1 of the bounding box_Top-left corner y1 of the bounding box_Bottom-right corner x2 of the bounding box_Bottom-right corner y2 of the bounding box". It should be noted that this naming convention is not unique; it only needs to reflect the pedestrian ID, video name, bounding box coordinates, and current time.

[0112] The specific implementation of step (4) may include the following sub-steps:

[0113] (4.1) Based on the map adjacency matrix, directly obtain x cameras (x>=1) that are geographically adjacent to the camera corresponding to the video, and load these x cameras C1, C2…C…in sequence. k From (T) f +T a ) to (T f +T a +ΔT) video clip, where T a The simplified search duration is obtained based on the map adjacency matrix, where ΔT is the search duration. Each video segment and its corresponding start and end times are saved.

[0114] Specifically, based on the map adjacency matrix input from the camera's geographical location information, k cameras (k>=1) that are geographically adjacent to the cameras in step (2) are obtained, and these k cameras C1, C2...C1 are loaded sequentially. k From (T) f +T a ) to (T f +T a +ΔT) video segments (ΔT is the search duration), save each video segment and name it "video name_(T)". f +T a )_(Tf +T a +ΔT)".T a To simplify the search time obtained from the map adjacency matrix, the value in the i-th row and j-th column of adjacency matrix A is obtained from the map adjacency matrix values. ij If A ij =0, then camera C i and C j If A is not geographically adjacent ij If > 0, then camera C i and C j Geographically adjacent, A ij For pedestrians from camera C i Walk to camera C j A conservative estimate of the shortest time required (reflecting the geographical distance between cameras).

[0115] The process of creating the map adjacency matrix can be as follows: Based on the camera location and road information in the map, first determine whether any two cameras are connected by a road that does not pass through other cameras. If so, they are considered adjacent; otherwise, they are not adjacent. Second, obtain the actual distance between two adjacent cameras based on the map, and create the map adjacency matrix based on the above two conditions: if they are adjacent, record it as the distance between the two cameras; if they are not adjacent, record it as 0.

[0116] (4.2) Perform pedestrian detection using the first target detection algorithm on the video frames in each video segment, resize the image within the obtained rectangle to match the input size of the re-identification network in step (3), adjust the average brightness of the image to the same average brightness level as the video frame in step (1), and save the image and the corresponding time and coordinates.

[0117] Specifically, pedestrian detection is performed every 10 frames using Yolov5. The image within the resulting bounding box is resized to 64*128, and the average brightness of the image is adjusted to the same average brightness level as the target image in the multi-query dataset created in step (1). A gallery dataset is created to save the images. The naming rule for saving images in this step is "video name_current time_x1 of the top left corner of the bounding box_y1 of the top left corner of the bounding box_x2 of the bottom right corner of the bounding box_y2 of the bottom right corner of the bounding box". It should be noted that this naming rule is not unique, as long as it reflects the video name, bounding box coordinates, and current time. In addition, it should be noted that "pedestrian detection is performed on each video frame in the video segment using the first target detection algorithm" means that pedestrian detection is performed on each video frame in the video segment every predetermined number of frames. In this embodiment, the predetermined number of frames is set to 10 frames, which can be set according to the actual situation.

[0118] (4.3) Input the image saved in step (4.2) and the image within the rectangular frame of the target to be tracked saved in step (2) into the re-identification network to determine whether the target to be tracked exists in the image;

[0119] Specifically, the network inputs a series of images of the target to be tracked and a single image for comparison. In this embodiment, the average value of each dimension of the 512-dimensional vector obtained from the series of images is used to obtain another 512-dimensional vector, which is then used to calculate the Euclidean distance with the other image.

[0120] (4.4) If there is, return to step (2); if not, expand the search time range and return to step (4.1) to search again. If the search continues to fail, continuously expand the search time range until the search is successful or all video segments are searched, thereby obtaining the complete trajectory information of the target to be tracked.

[0121] Specifically, if the target is successfully re-identified, return to step (2); otherwise, the search time needs to be extended further, and the search time should be adjusted to (T). f +T a +ΔT) to (T) f +T a +ΔT+ΔT1), return to step (4.1) to search again. If the search continues to fail, the search time will be increased until the search is successful or all video segments are obtained, so as to obtain the complete trajectory information of the target to be tracked, where ΔT1 is the search time.

[0122] The specific implementation of step (5) may include the following sub-steps:

[0123] (5.1) Read the complete trajectory information of the target to be tracked in sequence to obtain the two cameras C1 and C2 that pass by in sequence;

[0124] Specifically, the system reads two adjacent rows from the record database in sequence, where the first character of each row, separated by a space, is the camera name.

[0125] (5.2) Use the already marked map to obtain the coordinate positions P1 and P2 of the two cameras on the map, and perform binarization on the map to obtain road information;

[0126] Specifically, the system uses a pre-marked map to read the coordinates of the two cameras, P1 and P2, on the map. Both P1 and P2 are on the corresponding roads. Then, the map is binarized, and the pixel value of the road is 1, while the pixel value of the non-road is 0, in order to obtain road information.

[0127] (5.3) Use the growth algorithm to obtain the most reasonable route between points P1 and P2 in the binarized map, record the coordinates of each point in the route, and then draw the route on the original map.

[0128] Specifically, the region growing algorithm is used to obtain the most reasonable route between points P1 and P2 within the binarized map, and the coordinates of each point along the route are recorded. The route is then drawn onto the original map. The formula for the region growing algorithm is:

[0129] Grow(P x,y =InRoad(P x,y+1 ,P x，y-1 ,P x+1,y ,P x-1,y )

[0130] Where x and y are the coordinates of point P, and the InRoad function determines point P from the map. x,y If a point is located in one of the four directions (up, down, left, right), it is considered to be within the road. If it is, it is retained and added to the growth set. The growth function Grow is used iteratively until the growth set contains P2.

[0131] (5.4) Repeat steps (5.1) to (5.3) until all the recorded trajectory information of the target to be tracked is read.

[0132] Implementation Examples

[0133] An implementation example of the present invention was conducted on a machine equipped with an Intel Core i7-9750H central processing unit, an NVIDIA GTX1650 graphics processor, and 16GB of memory, and the experimental results shown in the attached figures were obtained. The system automatically plots the map trajectory of the person to be tracked, extracts video clips of the person appearing in the videos from various cameras, performs risk assessment on other pedestrians by detecting their masks, and saves images of pedestrians in close contact with the target being tracked.

[0134] like Figure 2 As shown, this system uses the SSD algorithm to detect masks, classifies pedestrians who have been in close contact with the target to be tracked into risk levels, and saves the classifications in the contact database.

[0135] like Figure 3 As shown, this system automatically generates the travel trajectory of the target to be tracked, which greatly reduces manual work, improves the accuracy of epidemiological investigation, and has high efficiency. It takes about 60 minutes to process a video with a total duration of 156 minutes, which shortens the epidemiological investigation time.

[0136] Corresponding to the aforementioned embodiments of the contact tracing assistance method based on machine vision and deep learning, this application also provides embodiments of a contact tracing assistance device based on machine vision and deep learning.

[0137] Figure 4 This is a block diagram illustrating an epidemiological investigation assistance device based on machine vision and deep learning, according to an exemplary embodiment. (Refer to...) Figure 4 The device may include:

[0138] Target acquisition module 21: acquires video and video frames in which the target is not occluded, performs pedestrian detection on the video frames to obtain several selection boxes, selects the corresponding selection box as the selection box for the target to be tracked based on the received user instruction, and saves the target image and the corresponding time in the corresponding selection box.

[0139] Single-lens target tracking module 22: Based on the target image and corresponding time stored in the target acquisition module, the target is tracked forward and backward respectively to obtain all video frames in which the target appears in the video, record the trajectory information of the target and save the image of the target within the rectangle, the rectangle coordinates and time corresponding to each video frame;

[0140] Mask detection and risk personnel classification and storage module 23: Pedestrian detection and mask detection are performed on all video frames in which the target to be tracked appears in the video, and then the classification and storage are performed through a re-identification algorithm;

[0141] Cross-camera target re-identification module 24: After the target to be tracked disappears from the video, it searches the video of the corresponding camera in the map adjacency matrix representing the geographical location of the camera to see if the target to be tracked exists, thereby obtaining the complete trajectory information of the target to be tracked.

[0142] Target trajectory drawing module 25: Draws the trajectory of the target to be tracked on the map based on the complete trajectory information of the target to be tracked.

[0143] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0144] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0145] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the above-described machine vision and deep learning-based epidemiological investigation assistance method.

[0146] Accordingly, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the above-described machine vision and deep learning-based epidemiological investigation assistance method.

[0147] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0148] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for assisting epidemiological investigation based on machine vision and deep learning, characterized in that, include: (1) Acquisition of target to be tracked: acquire video and video frames in which the target is not occluded, perform pedestrian detection on the video frames to obtain several selection boxes, select the corresponding selection box as the selection box of the target to be tracked by the received user instruction, and save the image of the target to be tracked in the corresponding selection box and the corresponding time. (2) Single-lens target tracking: Based on the target image to be tracked and the corresponding time saved in step (1), the target to be tracked is tracked forward and backward respectively to obtain all video frames in which the target to be tracked appears in the video, record the trajectory information of the target to be tracked and save each video frame and the corresponding rectangular frame image, rectangular frame coordinates and time of the target to be tracked; (3) Mask detection and classification and storage of at-risk personnel: all video frames in which the target to be tracked appears in the video are subjected to pedestrian detection and mask detection, and then classified and stored by a re-identification algorithm; (4) Cross-camera target re-identification: After the target to be tracked disappears from the video, the target to be tracked is searched in the video of the corresponding camera in the map adjacency matrix representing the geographical location of the camera to obtain the complete trajectory information of the target to be tracked. (5) Target trajectory drawing: Based on the complete trajectory information of the target to be tracked, draw the trajectory of the target to be tracked on the map; Step (3) includes the following sub-steps: (3.1) In all video frames in which the target to be tracked appears, the second target detection algorithm is used to detect whether the corresponding pedestrians are wearing masks in the images within the rectangular frames other than the target to be tracked, and then risk classification is performed; (3.2) Input the images inside the rectangles except the target rectangle to be tracked into the re-identification network to determine whether the corresponding pedestrian has been identified. If the corresponding pedestrian has been identified, save the images inside the rectangles, the corresponding pedestrian number, time, and rectangle coordinates to the corresponding dataset. Otherwise, create a new dataset and save the images inside the rectangles, the corresponding pedestrian number, time, and rectangle coordinates to the new dataset. Step (4) includes the following sub-steps: (4.1) Based on the map adjacency matrix, directly obtain x cameras (x>=1) that are geographically adjacent to the camera corresponding to the video, and load these x cameras sequentially. , … from arrive Video clips, in which, The simplified search time is obtained based on the map adjacency matrix. For the search duration, save each video segment and its corresponding start and end times; (4.2) Perform pedestrian detection on the video frames in each video segment using the first target detection algorithm, resize the image within the obtained rectangle to match the input size of the re-identification network in step (3), adjust the average brightness of the image to the same average brightness level as the video frame in step (1), and save the image and the corresponding time and coordinates; (4.3) Input the image saved in step (4.2) and the image within the rectangular frame of the target to be tracked saved in step (2) into the re-identification network to determine whether the target to be tracked exists in the image; (4.4) If found, return to step (2); if not, expand the search time range and return to step (4.1) to search again. If the search continues to fail, continuously expand the search time range until the search is successful or all video segments are found, thereby obtaining the complete trajectory information of the target to be tracked. This represents the duration of the search.

2. The method according to claim 1, characterized in that, Step (1) includes the following sub-steps: (1.1) After obtaining a video frame in which the target to be tracked is not occluded, the first target detection algorithm is used to perform pedestrian recognition on the video frame, and the recognition result is marked with a box in the video frame to generate several selection boxes for the user to select the target to be tracked. (1.2) Select the corresponding selection box of the received user instruction as the selection box of the target to be tracked, and save the image of the target to be tracked in the corresponding selection box and the corresponding time.

3. The method according to claim 1, characterized in that, Step (2) includes the following sub-steps: (2.1) Obtain the video from the database. Images at different times are used for pedestrian recognition, resulting in k bounding boxes. , … (k>=1), perform overlap matching between all the bounding boxes on the image and the selection box of the target to be tracked to obtain the bounding box corresponding to the target. And record the relevant coordinate information; (2.2) From Start loading video frames forward and perform pedestrian detection on each frame using the first object detection algorithm, with a time of [time value missing]. , obtain the i-shaped rectangle , … (i>=1), and simultaneously, a multi-target tracking algorithm is used to track the target until the bounding box of the target disappears in the video. The forward trajectory information and disappearance time of the target in the video are recorded. Save each video frame along with the corresponding image within the bounding box of the target to be tracked, the bounding box coordinates, and the time. (2.3) From Start loading video frames sequentially and perform pedestrian detection on each frame using the first object detection algorithm, with a time of [time value missing]. Frame, obtain j rectangular boxes , … (j>=1), and simultaneously, a multi-target tracking algorithm is used to track the target until the bounding box of the target disappears from the video. The backward trajectory information and disappearance time of the target within the video are recorded. Each video frame, along with the corresponding image within the bounding box of the target to be tracked, the bounding box coordinates, and the time, is saved.

4. The method according to claim 3, characterized in that, Step (2) further includes the following sub-steps: (2.4) For the images within the rectangular bounding box of the target to be tracked saved in steps (2.2) and (2.3), the deletion time is less than or greater than Images are used to remove incomplete images that may appear and disappear when the target to be tracked appears.

5. The method according to claim 1, characterized in that, Step (5) includes the following sub-steps: (5.1) Read the complete trajectory information of the target to be tracked in sequence to obtain the two cameras that pass by in sequence. ; (5.2) Use the already marked map to obtain the coordinates of the two cameras on the map. The map is then binarized to obtain road information. (5.3) Using a growing algorithm to obtain data within the binarized map and Find the most reasonable route between two points, record the coordinates of each point along the route, and then draw the route on the original map. (5.4) Repeat steps (5.1) to (5.3) until all the recorded trajectory information of the target to be tracked is read.

6. An apparatus for an epidemiological investigation assistance method based on machine vision and deep learning as described in claim 1, characterized in that, include: Target acquisition module: acquires video and video frames in which the target is not occluded, performs pedestrian detection on the video frames to obtain several selection boxes, selects the corresponding selection box as the selection box for the target to be tracked based on the received user instruction, and saves the image of the target to be tracked in the corresponding selection box and the corresponding time. Single-lens target tracking module: Based on the target image and corresponding time stored in the target acquisition module, the target is tracked forward and backward respectively to obtain all video frames in which the target appears in the video, record the trajectory information of the target and save the image of the target within the rectangle, the rectangle coordinates and time corresponding to each video frame; Mask detection and at-risk personnel classification and storage module: Pedestrian detection and mask detection are performed on all video frames in which the target to be tracked appears in the video, and then the classification and storage are performed through a re-identification algorithm; Cross-camera target re-identification module: After the target to be tracked disappears from the video, the module searches the video of the corresponding camera in the map adjacency matrix, which represents the geographical location of the camera, to see if the target to be tracked exists, thereby obtaining the complete trajectory information of the target to be tracked. Target trajectory drawing module: Based on the complete trajectory information of the target to be tracked, draw the trajectory of the target to be tracked on the map.

7. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-5.

Images