Multi-objective hybrid preference method, apparatus, electronic device, and storage medium

Abstract

The application relates to a multi-target hybrid optimization method, device, electronic device and storage medium, wherein the method comprises: acquiring video frame sequences in a monitoring video frame by frame, and performing target detection and tracking on targets in each video frame sequence to determine target tracking results of the targets; using a target auxiliary algorithm to process the target tracking results to obtain target auxiliary information; and using a preset optimization module to independently and / or associatively optimize the target tracking results and the target auxiliary information for each target to obtain an optimization result. Through the application, the problem that, in the related art, target scores are used as the only evaluation condition for target optimization, resulting in low optimization precision, is solved, and the independent and / or associative optimization of multiple targets is realized by using target auxiliary information in combination with target tracking results under complex scene video monitoring, so that the optimization precision is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to multi-objective hybrid optimization methods, apparatus, electronic devices, and storage media. Background Technology

[0002] With the continuous development of computer vision technology, its applications are becoming increasingly widespread. Among these applications, target recognition in computer vision has received particular attention.

[0003] Currently, the preferred approach for target selection in computer vision involves acquiring a sequence of several input video frames, detecting the target to be tracked in the current frame using an SSD detector, tracking the target using a multi-target tracking algorithm based on the Hungarian algorithm, associating, assigning values, and marking the target based on the marked target in the previous frame, and finally scoring the candidate key target images in the current frame to update the preferred target image. The drawback of this approach is that using target score as the sole evaluation criterion for target selection is insufficient in access control gate video surveillance applications, leading to low selection accuracy.

[0004] There is currently no effective solution to the problem that using target score as the sole evaluation criterion for target optimization in related technologies leads to low optimization accuracy. Summary of the Invention

[0005] This embodiment provides a multi-objective hybrid optimization method, apparatus, electronic device, and storage medium to solve the problem of low optimization accuracy caused by using target scores as the sole evaluation condition for target optimization in related technologies.

[0006] Firstly, this embodiment provides a multi-objective hybrid optimization method, including:

[0007] The video frame sequence in the surveillance video is acquired frame by frame, and the target in each video frame sequence is detected and tracked to determine the target tracking result of each target.

[0008] The target tracking results are processed using a target-assisted algorithm to obtain target-assisted information;

[0009] Using a preset optimization module, the target tracking results and the target auxiliary information are optimized independently and / or in association for each target to obtain an optimal result.

[0010] In some embodiments, the target detection and tracking of targets in each video frame sequence to determine the target tracking result for each target includes:

[0011] Using object detection algorithms, object detection is performed on the video frames corresponding to each frame sequence to detect each object;

[0012] The target tracking algorithm is used to track each target and obtain the target tracking results for each target.

[0013] In some embodiments, the target detection algorithm is one of the ACF algorithm, DMP algorithm, and CNN algorithm.

[0014] In some embodiments, the target tracking algorithm is one of the FHOG algorithm and the GoTurn algorithm.

[0015] In some embodiments, the target-assisted algorithm is one or more of the following: a target quality scoring algorithm, a target orientation angle algorithm, and a liveness algorithm.

[0016] In some embodiments, the step of using a preset optimization module to perform independent and / or correlated optimization of the target tracking results and the target auxiliary information to obtain optimized results includes:

[0017] The target tracking results and the target auxiliary information are distributed to the preferred sub-modules in the preset preferred module;

[0018] When the optimization submodule is a multi-target association optimization submodule, the target tracking result and the target auxiliary information are subjected to multi-target hybrid optimization to obtain the multi-target optimization result;

[0019] When the optimization submodule is a single-target independent optimization submodule, single-target optimization is performed on the target tracking result and the target auxiliary information to obtain the single-target optimization result;

[0020] The optimal result is determined based on the multi-objective optimization results and / or the single-objective optimization results.

[0021] In some embodiments, the step of performing single-target optimization on the target tracking result and the target auxiliary information to obtain a single-target optimization result includes:

[0022] Target analysis is performed on the target tracking results and the target auxiliary information to obtain effective target information;

[0023] Traverse the valid target information;

[0024] According to a preset strategy, obtain the operation code corresponding to the valid target information;

[0025] The effective target information of the target is operated on according to the opcode to obtain a single target optimization result.

[0026] In some embodiments, the multi-target hybrid optimization of the target tracking result and the target auxiliary information to obtain a multi-target optimization result includes:

[0027] The target tracking results and the target auxiliary information are distributed to the single-target independent selection unit in the multi-target association selection submodule;

[0028] In the single-target independent optimization unit, the target tracking result and the target auxiliary information are optimized as a single target, and the single-target optimization result is optimized as a related optimization according to a preset association optimization logic to obtain a multi-target optimization result.

[0029] Secondly, this embodiment provides a multi-target hybrid optimization device, including: an acquisition module, a processing module, and a target optimization module;

[0030] The acquisition module is used to acquire video frame sequences in the surveillance video frame by frame, and to perform target detection and tracking on the targets in each video frame sequence in order to determine the target tracking results of each target.

[0031] The processing module is used to process the target tracking results using a target assistance algorithm to obtain target assistance information;

[0032] The target optimization module is used to perform independent and / or correlated optimization of each target based on the target tracking results and the target auxiliary information using a preset optimization module, so as to obtain an optimization result.

[0033] Thirdly, this embodiment provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-objective hybrid optimization method described in the first aspect above.

[0034] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the multi-objective hybrid optimization method described in the first aspect above.

[0035] Compared with related technologies, the multi-target hybrid optimization method, apparatus, electronic device, and storage medium provided in this embodiment acquire video frame sequences from the surveillance video frame by frame, and perform target detection and tracking on the targets in each video frame sequence to determine the target tracking results of each target; use a target-assisted algorithm to process the target tracking results to obtain target-assisted information; and use a preset optimization module to perform independent and / or associated optimization of each target based on the target tracking results and target-assisted information to obtain the optimization result. This solves the problem of low optimization accuracy caused by using target scores as the sole evaluation condition for target optimization in related technologies, and realizes the independent and / or associated optimization of multiple targets by combining target-assisted information with target tracking results in complex video surveillance scenarios, thereby improving the optimization accuracy.

[0036] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 This is a hardware structure block diagram of a terminal device for a multi-objective hybrid optimization method provided in an embodiment of this application;

[0039] Figure 2 This is a flowchart of a multi-objective hybrid optimization method provided in an embodiment of this application;

[0040] Figure 3 This is a structural block diagram of a target preferred module provided in an embodiment of this application;

[0041] Figure 4 This is a structural block diagram of a target preferred module provided in an embodiment of this application;

[0042] Figure 5 This is a structural block diagram of a target preferred module provided in an embodiment of this application;

[0043] Figure 6 yes Figure 2 Flowchart of step S230;

[0044] Figure 7 This is a schematic diagram of the internal steps of the management module of a preferred embodiment provided in this application;

[0045] Figure 8 This is a structural block diagram of a multi-objective hybrid optimization device provided in an embodiment of this application.

[0046] In the diagram: 210, Acquisition Module; 220, Processing Module; 230, Target Optimization Module; 31, Optimization Scheme Management Module; 32, Optimization Submodule Interface; 33, Multi-Target Association Optimization Submodule; 34, Single-Target Independent Optimization Submodule. Detailed Implementation

[0047] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal of the multi-objective hybrid optimization method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0050] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-objective hybrid optimization method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0052] This embodiment provides a multi-objective hybrid optimization method. Figure 2 This is a flowchart of the multi-objective hybrid optimization method in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0053] Step S210: Acquire the video frame sequence in the monitoring video frame by frame, and perform target detection and tracking on the target in each video frame sequence to determine the target tracking result of each target;

[0054] Step S220: The target tracking result is processed using a target-assisted algorithm to obtain target-assisted information;

[0055] Step S230: Using a preset optimization module, perform independent and / or correlated optimization of each target based on the target tracking results and target auxiliary information to obtain the optimization results.

[0056] It should be noted that multi-target hybrid optimization refers to the association optimization of multiple types of targets, as well as a hybrid optimization that combines association optimization or individual optimization; for example, association optimization includes: face, body, and license plate association optimization; face and body association optimization; individual optimization includes: license plate individual optimization, etc. The surveillance video is processed using a frequency processing algorithm to obtain a video frame sequence; a video frame sequence refers to the sequence of images obtained from the surveillance video. Each video frame sequence may contain a target, therefore, the target to be tracked in each video frame sequence is first detected, and then the target is tracked to obtain the target tracking result.

[0057] To improve the accuracy of the selection, this embodiment adds target auxiliary information as a reference. This target auxiliary information includes, but is not limited to, target quality score information, target azimuth angle information, and liveness information. The target auxiliary information is determined by a specific target auxiliary algorithm; different algorithms will result in different target auxiliary information. For example, if the target auxiliary algorithm is a target quality score algorithm, then the target auxiliary information will be the target quality score information. In other embodiments, the target auxiliary information can be one or more types, and there is no limitation on this.

[0058] The preferred module is preset, such as Figure 3 As shown, the optimization module may include: an optimization scheme management module 31, an optimization submodule interface 32, a multi-target associated optimization submodule 33, and a single-target independent optimization submodule 34. The optimization scheme management module 31 manages the optimization submodules (multi-target associated optimization submodule 33 and single-target independent optimization submodule 34) through the optimization submodule interface 32, distributing target tracking results and target auxiliary information to the submodules for corresponding processing. For example: distributing to the multi-target associated optimization submodule 33 for associated optimization; distributing to the single-target independent optimization submodule 34 for independent optimization; or distributing to both the multi-target associated optimization submodule 33 and the single-target independent optimization submodule 34 for both associated and independent optimization. The optimization module can be enabled according to the actual application situation; for example, if only independent optimization is involved, then only the single-target independent optimization submodule 34 is enabled. The multi-target associated optimization submodule 33 can be composed of multiple single-target independent optimization submodules 34, with the associated optimization logic controlling the associated optimization of the multiple single-target independent optimization submodules 34 within the multi-target associated optimization submodule 33. In other embodiments, the preferred module can also be set according to the actual application. For example: Figure 4 As shown, the optimization module includes: an optimization scheme management module 31, an optimization submodule interface 32, and a multi-target association optimization submodule 33. For example... Figure 5 As shown, the optimization module includes: optimization scheme management module 31, optimization submodule interface 32, and single-target independent optimization submodule 34; the specific structure of the optimization module is not restricted.

[0059] In existing technical solutions, most focus on independent optimization of single-type targets, neglecting the associated optimization of multiple target types, or hybrid optimization combining associated and individual optimization. Specifically, the optimization scheme uses target score as the sole evaluation criterion. However, a single target score evaluation criterion is insufficient for access control gate video surveillance applications, where the largest face in the video should be selected as the optimal target. Using a single target score evaluation criterion leads to low optimization accuracy. In contrast, this application, based on target tracking results, combines target auxiliary information to perform independent and / or associated optimization of multiple targets, improving optimization accuracy. Furthermore, it solves the technical problem of requiring associated and hybrid optimization of multiple target types in complex video surveillance scenarios. For example, in video surveillance of large scenes such as roads, squares, and stations, associated target optimization and the coexistence of associated and independent optimization are required. The method proposed in this embodiment can achieve both associated and independent target optimization, exhibiting strong reusability and scalability.

[0060] The following is a detailed explanation of each step:

[0061] In some embodiments, step S210, which involves detecting and tracking targets in each video frame sequence to determine the target tracking result for each target, includes the following steps:

[0062] Using object detection algorithms, object detection is performed on the video frames corresponding to each frame sequence to detect each object;

[0063] Using target tracking algorithms, each target is tracked, and the target tracking results for each target are obtained.

[0064] The target detection algorithm includes, but is not limited to, one of the ACF, DMP, and CNN algorithms. When performing target detection on video frames corresponding to each frame sequence, any one of the ACF, DMP, and CNN algorithms can be used to detect each target. The target tracking algorithm includes, but is not limited to, one of the FHOG and GoTurn algorithms. When tracking each target, the FHOG or GoTurn algorithm can be used to obtain the target tracking result for each target. In this embodiment, no restrictions are placed on the target detection and target tracking algorithms.

[0065] In some embodiments, the target assistance algorithm includes, but is not limited to, one or more of the following: target quality scoring algorithm, target orientation angle algorithm, and liveness detection algorithm. When processing the target tracking results, one or more of the following algorithms can be used to obtain target assistance information: target quality scoring algorithm, target orientation angle algorithm, and liveness detection algorithm.

[0066] In some of these embodiments, such as Figure 6 As shown, step S230 includes the following steps;

[0067] Step S231: Distribute the target tracking results and target auxiliary information to the preferred sub-modules in the preset preferred module;

[0068] Step S232: When the optimization submodule is a multi-target association optimization submodule, multi-target hybrid optimization is performed on the target tracking results and target auxiliary information to obtain multi-target optimization results;

[0069] Step S233: When the optimization submodule is a single-target independent optimization submodule, perform single-target optimization on the target tracking results and target auxiliary information to obtain the single-target optimization result;

[0070] Step S234: Determine the optimal result based on the multi-objective optimization results and / or the single-objective optimization results.

[0071] The flowchart illustrating the internal steps of the preferred solution management module is as follows: Figure 7 As shown, it mainly includes the following steps:

[0072] S51. Obtain the target tracking results and target auxiliary information in the input frame;

[0073] S52. Input the acquired target tracking results and target auxiliary information into the optimal solution management module;

[0074] S53, the optimal solution management module distributes the input information to one or more of its sub-modules;

[0075] S54. If the optimization submodule is a single-objective independent optimization submodule, then proceed to step S233; if the optimization submodule is a multi-objective associated optimization submodule, then proceed to step S232.

[0076] S55. Summarize the best results.

[0077] The preferred submodule interface is a functional interface for the preferred submodule, and does not contain business logic.

[0078] Specifically, the optimization module can be configured according to the actual application. The optimization scheme management module within the optimization module can then distribute target tracking results and target auxiliary information to the corresponding optimization sub-modules (multi-target associated optimization sub-modules and single-target independent optimization sub-modules). For example, if the optimization sub-module only has a multi-target associated optimization sub-module, then the target tracking results and target auxiliary information will be distributed to it. If the optimization sub-module only has a single-target independent optimization sub-module, then the target tracking results and target auxiliary information will be distributed to it. If the optimization sub-module has both multi-target associated optimization sub-modules and single-target independent optimization sub-modules, then the target tracking results and target auxiliary information will be distributed to both of these sub-modules.

[0079] In the single-target independent optimization submodule, single-target optimization is performed on the target tracking results and target auxiliary information to obtain the single-target optimization result; in the multi-target associated optimization submodule, multi-target hybrid optimization is performed on the target tracking results and target auxiliary information to obtain the multi-target optimization result; finally, the optimization result is determined based on the multi-target optimization result and / or the single-target optimization result.

[0080] The single-target independent optimization submodule inherits the optimization submodule interface and primarily implements the independent optimization logic for a single type of target. It can also optimize one main target plus multiple auxiliary targets, where the auxiliary targets are merely supplementary information to the main target and are output together with the main target in the same frame during optimization output. The single-target independent optimization submodule is suitable for scenarios where single-type target optimization is performed independently, such as face recognition gate scenarios and thermal imaging scenarios. It is also suitable for scenarios where multiple single-type target optimizations coexist, such as scenarios where independent optimization of pedestrians and vehicles coexists.

[0081] The multi-target association and optimization submodule inherits the optimization submodule interface and mainly implements association and optimization between various types of targets. It includes multiple independent single-target optimization modules. The association output can be: after one independent single-target optimization module has an optimization result, it triggers other independent single-target optimization modules to output their optimization results. The specific association logic is not limited. The multi-target association and optimization submodule is suitable for various target association and optimization scenarios; for example, face and body association optimization in pedestrian video surveillance, and license plate and vehicle association optimization in road video surveillance.

[0082] The single-target independent optimization submodule, combined with the multi-target associated optimization submodule, is suitable for scenarios where associated optimization and independent optimization coexist. For example, in road monitoring, face and body associated optimization and vehicle optimization coexist. The single-target independent optimization submodule combined with the multi-target associated optimization submodule is also suitable for various scenarios where associated optimization coexists, such as face and body associated optimization and license plate and vehicle associated optimization coexist.

[0083] In some embodiments, step S233, which involves performing single-target optimization on the target tracking results and target auxiliary information to obtain a single-target optimization result, includes:

[0084] Target analysis is performed on the target tracking results and target auxiliary information to obtain effective target information;

[0085] Traverse the valid target information;

[0086] According to the preset strategy, obtain the operation code corresponding to the valid target information;

[0087] The operation code is used to operate on the valid target information of the target to obtain the single target optimization result.

[0088] Specifically, before performing target analysis on the target tracking results and target auxiliary information, data preprocessing can be performed on the target tracking results and target auxiliary information. Valid targets depend on the specific optimization scheme of the single-target independent optimization submodule. For example, in a face optimization scheme, the valid target is a face. Valid target information includes, but is not limited to, the following: target tracking ID, target tracking status, target quality score, target bounding box position, target orientation angle, target liveness status, etc. Traversing valid target information can involve comparing each valid target's face with those in the historical cache. Preset strategies include, but are not limited to, the following: if a target appears consecutively for N frames, one frame is optimized; if a target appears and disappears, only one frame is optimized; if a target triggers a certain alarm, one frame is optimized, etc. Opcodes include, but are not limited to, the following: NEW adds to the historical best cache; UPDATE replaces information in the historical best cache; EXPORT outputs the optimization result; DELETE deletes from the historical best cache. All of the above opcodes can exist simultaneously; for example, updating the cache and outputting the optimization result. After obtaining the opcode, operations are performed on the valid target information of the target according to the opcode to obtain the single-target optimization result. For example, operating on the information of the current target ID.

[0089] In some embodiments, step S232, which involves performing multi-target hybrid optimization on the target tracking results and target auxiliary information to obtain multi-target optimization results, includes:

[0090] Distribute the target tracking results and target auxiliary information to the single-target independent selection units in the multi-target association selection submodule;

[0091] In the single-target independent optimization unit, the target tracking results and target auxiliary information are optimized for a single target, and the single-target optimization results are optimized for a multi-target optimization result according to the preset association optimization logic.

[0092] Specifically, the target tracking results and target auxiliary information are distributed to the single-target independent selection units in the multi-target association and selection submodule. In each single-target independent selection unit, the relevant steps as described in step S230 are executed. Then, association and selection are performed on each single-target selection result, executing the following steps... Figure 7 The relevant steps are then followed, and finally, the associated optimization logic is executed to obtain the multi-objective optimization result. This associated logic includes, but is not limited to, the coordinated execution of one or more other single-objective independent optimization sub-modules to output the optimized objective.

[0093] In video surveillance of large-scale scenarios such as roads, squares, and stations, target association optimization is required, as well as situations where association optimization and independent optimization coexist. The method proposed in this embodiment can achieve target association optimization, as well as the coexistence of association optimization and independent optimization, and has strong reusability and scalability.

[0094] The present embodiment will now be described and illustrated through preferred embodiments.

[0095] In the face and body association optimization service in the human video surveillance scenario, the face optimization logic is: one frame is optimized every 24 frames, and the association optimization logic is: when the face optimization is output, the human body associated with the current face is also optimized.

[0096] The specific implementation steps are as follows:

[0097] Y10, Target Detection Algorithm and Target Tracking Algorithm: Select Face and Human Body Detection Algorithm and Face and Human Body Correlation Tracking Algorithm;

[0098] Y11. The target-assisted algorithm selects face quality scoring algorithm, face angle algorithm, face liveness algorithm and human body quality scoring algorithm;

[0099] Y12, The optimal solution management module includes a face and body association optimization submodule (multi-target association optimization submodule);

[0100] Y13, the face and body association optimization submodule includes a face independent optimization submodule (a single-target independent optimization submodule for faces) and a body independent optimization submodule (a single-target independent optimization submodule for bodies);

[0101] Y14. The association logic in the face and body association optimization submodule is as follows: when the face optimization is output, the linked body independent optimization module outputs the human body with the same face ID.

[0102] Y15, The implementation process of the independent face selection submodule is as follows:

[0103] a) Input the preprocessed target tracking results and target auxiliary information. The target tracking results are for extracting only faces, while simultaneously extracting target auxiliary information (face quality score, face angle, and face liveness). Filter out non-live faces, faces with pitch, yaw, and roll angles greater than 45 degrees, and faces with a face quality score less than 15.

[0104] b) Iterate through all the extracted valid face information and compare each one with the valid face information with the same ID in the historical cache.

[0105] c) Obtain the operation code. If the current face ID is not found in the historical cache, the operation code is NEW; if the current face quality score is greater than that in the historical cache, the operation code is UPDATE; if the face appears consecutively for 24 frames, the operation code is EXPORT; if the face tracking status is DELETE, the operation code is DELETE. All of these operation codes can coexist.

[0106] d) Based on the operation code output in step c) of the face independent selection submodule above, perform the following operations: If the operation code contains NEW, add the current face information to the cache; if the operation code contains UPDATE, update the current face information to the cache; if the operation code contains EXPORT, output the information of the face ID in the historical cache; if the operation code contains DELETE, delete the information of the current face ID from the cache.

[0107] e) Summarize the best output results.

[0108] Y16, The implementation process of the human body independent optimization submodule is as follows:

[0109] a) Input the preprocessed target tracking results and target auxiliary information. The target tracking results extract only the human body, while simultaneously extracting target auxiliary information (human quality score).

[0110] b) Iterate through all the extracted valid human body information and compare each one with the valid human body information with the same ID in the historical cache.

[0111] c) Obtain the operation code. If the current human body ID is not found in the historical cache, the operation code is NEW; if the current human body quality score is greater than that in the historical cache, the operation code is UPDATE; if the human body tracking status is DELETE, the operation code is DELETE. These operation codes can coexist. The human body optimization submodule here does not automatically perform optimization. Therefore, the EXPORT operation code is not automatically generated and needs to be associated with the optimization module to assist in the optimization output.

[0112] d) Based on the operation code output in step c) of the above human body independent selection submodule, perform the following operations: if the operation code contains NEW, add the current human body information to the cache; if the operation code contains UPDATE, update the current human body information to the cache; if the operation code contains DELETE, delete the current human body ID information from the cache.

[0113] Y17. The implementation process of the face and body association optimization module is as follows:

[0114] a) Distribute target tracking results and target auxiliary information to the face-independent optimization submodule and the human-independent optimization submodule.

[0115] b) Determine if the face selection submodule outputs face selection results.

[0116] c) If a face selection result is output, the independent human body selection module will output the human body with the same ID as the face. If no human body with the same ID is found, the associated human body information will not be output.

[0117] d) Summarize the optimization results of the face independent optimization submodule and the human body independent optimization submodule.

[0118] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0119] This embodiment also provides a multi-objective hybrid optimization device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that implement predetermined functions. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0120] Figure 8 This is a structural block diagram of the multi-objective hybrid optimization device in this embodiment, as shown below. Figure 8As shown, the device includes: an acquisition module 210, a processing module 220, and a target optimization module 230; the acquisition module 210 is used to acquire video frame sequences in the surveillance video frame by frame, and to perform target detection and tracking on the targets in each video frame sequence to determine the target tracking results of each target; the processing module 220 is used to process the target tracking results using a target assistance algorithm to obtain target assistance information; the target optimization module 230 is used to use a preset optimization module to perform independent and / or correlated optimization of each target based on the target tracking results and target assistance information to obtain an optimization result.

[0121] This device solves the problem of low selection accuracy caused by using target rating as the sole evaluation condition for target selection in related technologies. It enables independent and / or associated selection of multiple targets in complex video surveillance scenarios by using target auxiliary information combined with target tracking results, thereby improving selection accuracy.

[0122] In some embodiments, the processing module 220 includes a target detection unit and a tracking unit; the target detection unit is used to perform target detection on the video frames corresponding to each frame sequence using a target detection algorithm to detect each target; the tracking unit is used to track each target using a target tracking algorithm to obtain the target tracking result of each target.

[0123] In some embodiments, the target selection module 230 is further configured to distribute the target tracking results and the target auxiliary information to a preset selection sub-module within the selection module;

[0124] When the optimization submodule is a multi-target association optimization submodule, the target tracking result and the target auxiliary information are subjected to multi-target hybrid optimization to obtain the multi-target optimization result;

[0125] When the optimization submodule is a single-target independent optimization submodule, single-target optimization is performed on the target tracking result and the target auxiliary information to obtain the single-target optimization result;

[0126] The optimal result is determined based on the multi-objective optimization results and / or the single-objective optimization results.

[0127] In some embodiments, the single-target independent optimization submodule in the target optimization module 230 is used to perform target analysis on the target tracking results and the target auxiliary information to obtain effective target information of the target;

[0128] Traverse the valid target information;

[0129] According to a preset strategy, obtain the operation code corresponding to the valid target information;

[0130] The effective target information of the target is operated on according to the opcode to obtain a single target optimization result.

[0131] In some embodiments, the multi-target association optimization submodule in the target optimization module 230 is used to distribute the target tracking results and the target auxiliary information to the single-target independent optimization unit in the multi-target association optimization submodule;

[0132] In the single-target independent optimization unit, the target tracking result and the target auxiliary information are optimized as a single target, and the single-target optimization result is optimized as a related optimization according to a preset association optimization logic to obtain a multi-target optimization result.

[0133] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0134] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0135] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0136] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0137] S1, acquire the video frame sequence in the monitoring video frame by frame, and perform target detection and tracking on the target in each video frame sequence to determine the target tracking result of each target;

[0138] S2, using a target-assisted algorithm, processes the target tracking results to obtain target-assisted information;

[0139] S3. Using a preset optimization module, the target tracking results and target auxiliary information are optimized independently and / or in association for each target to obtain the optimization result.

[0140] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0141] Furthermore, in conjunction with the multi-objective hybrid optimization method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the multi-objective hybrid optimization methods described in the above embodiments.

[0142] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0143] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0144] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A multi-objective hybrid optimization method, characterized in that, include: The video frame sequence in the surveillance video is acquired frame by frame, and the target in each video frame sequence is detected and tracked to determine the target tracking result of each target. The targets are of various types, including faces, bodies, and license plates; The target tracking results are processed using a target-assisted algorithm to obtain target-assisted information; The target auxiliary information includes target quality score information, target orientation angle information, and liveness information; Using a preset optimization module, the target tracking results and the target auxiliary information are optimized independently and / or correlated for each target to obtain an optimized result, specifically as follows: The target tracking results and the target auxiliary information are distributed to the preferred sub-modules in the preset preferred module; When the optimization submodule is a multi-target association optimization submodule, the target tracking result and the target auxiliary information are subjected to multi-target hybrid optimization to obtain the multi-target optimization result; When the optimization submodule is a single-target independent optimization submodule, single-target optimization is performed on the target tracking result and the target auxiliary information to obtain the single-target optimization result; The optimal result is determined based on the multi-objective optimization results and / or the single-objective optimization results.

2. The multi-objective hybrid optimization method according to claim 1, characterized in that, The process of detecting and tracking targets in each video frame sequence to determine the target tracking results for each target includes: Using object detection algorithms, object detection is performed on the video frames corresponding to each frame sequence to detect each object; The target tracking algorithm is used to track each target and obtain the target tracking results for each target.

3. The multi-objective hybrid optimization method according to claim 2, characterized in that, The target detection algorithm is one of the following: ACF algorithm, DMP algorithm, and CNN algorithm.

4. The multi-objective hybrid optimization method according to claim 2, characterized in that, The target tracking algorithm is one of the FHOG algorithm or the GoTurn algorithm.

5. The multi-objective hybrid optimization method according to claim 1, characterized in that, The target-assisted algorithm is one or more of the following: target quality scoring algorithm, target orientation angle algorithm, and liveness algorithm.

6. The multi-objective hybrid optimization method according to claim 1, characterized in that, The step of performing single-target optimization on the target tracking result and the target auxiliary information to obtain a single-target optimization result includes: Target analysis is performed on the target tracking results and the target auxiliary information to obtain effective target information; Traverse the valid target information; According to a preset strategy, obtain the operation code corresponding to the valid target information; The effective target information of the target is operated on according to the opcode to obtain a single target optimization result.

7. The multi-objective hybrid optimization method according to claim 1, characterized in that, The step of performing multi-target hybrid optimization on the target tracking result and the target auxiliary information to obtain a multi-target optimization result includes: The target tracking results and the target auxiliary information are distributed to the single-target independent selection unit in the multi-target association selection submodule; In the single-target independent optimization unit, the target tracking result and the target auxiliary information are optimized as a single target, and the single-target optimization result is optimized as a related optimization according to a preset association optimization logic to obtain a multi-target optimization result.

8. A multi-objective hybrid optimization device, characterized in that, include: Acquisition module, processing module, and target selection module; The acquisition module is used to acquire video frame sequences in the surveillance video frame by frame, and to perform target detection and tracking on the targets in each video frame sequence in order to determine the target tracking results of each target. The targets are of various types, including faces, bodies, and license plates; The processing module is used to process the target tracking results using a target assistance algorithm to obtain target assistance information; The target auxiliary information includes target quality score information, target orientation angle information, and liveness information; The target selection module is used to perform independent and / or correlated selection of each target based on the target tracking results and the target auxiliary information using a preset selection module, to obtain a selection result, specifically as follows: The target tracking results and the target auxiliary information are distributed to the preferred sub-modules in the preset preferred module; When the optimization submodule is a multi-target association optimization submodule, the target tracking result and the target auxiliary information are subjected to multi-target hybrid optimization to obtain the multi-target optimization result; When the optimization submodule is a single-target independent optimization submodule, single-target optimization is performed on the target tracking result and the target auxiliary information to obtain the single-target optimization result; The optimal result is determined based on the multi-objective optimization results and / or the single-objective optimization results.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the multi-objective hybrid preferred method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-objective hybrid optimization method according to any one of claims 1 to 7.

Images