A gait recognition method and device based on video instance segmentation

Abstract

The application provides a gait recognition method and device based on video instance segmentation, relates to the technical field of feature recognition, and collects gait video of a target person through a camera, inputs the gait video into a trained video instance segmentation model, and obtains a gait sequence of the target person; extracts gait features of the target person from the obtained gait sequence; compares the extracted gait features with a base library, and obtains a recognition object meeting a set threshold; that is, the gait sequence can be obtained through the trained video instance segmentation model in one step, so that the gait recognition step is simplified, and the model generalization is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of feature recognition, in particular to a gait recognition method and device based on video instance segmentation. BACKGROUND

[0002] Gait recognition is a kind of biological feature recognition means at a long distance without the cooperation of the object, aiming to recognize the identity of individuals, identify or detect the physiological, pathological and psychological characteristics according to the gait of people. Gait recognition can be performed under low image quality, without the cooperation of the recognized object, at a long distance, and is difficult to disguise and hide, which has obvious advantages compared with traditional biological feature recognition.

[0003] However, the gait recognition technology is currently complicated in steps and low in model generalization. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a gait recognition method and device based on video instance segmentation, electronic equipment and storage medium, which can simplify the gait recognition steps.

[0005] In a first aspect, the present application provides a gait recognition method based on video instance segmentation, which comprises the following steps:

[0006] A gait video of a target person is collected by a camera, and the gait video is input into a trained video instance segmentation model to obtain a gait sequence of the target person; wherein the gait sequence is a silhouette sequence, a key point sequence or a human body part sequence;

[0007] Gait features of the target person are extracted from the obtained gait sequence; wherein the method for extracting the gait features of the target person is different for the silhouette sequence, the key point sequence or the human body part sequence;

[0008] The extracted gait features are compared with a base library to obtain a recognized object that meets a set threshold.

[0009] In some embodiments, the silhouette sequence of the target person is obtained by the following method:

[0010] A pre-trained first video instance segmentation model trained by using an open source video instance segmentation dataset is obtained; wherein the first video instance segmentation model is constructed based on SeqFormer;

[0011] The segmentation supervision information in the open source video instance segmentation dataset is replaced so that all categories other than the target person are regarded as background;

[0012] fine-tune the pre-trained first video instance segmentation model by using the open source video instance segmentation dataset after the segmentation supervision information is replaced, to obtain a trained first video instance segmentation model;

[0013] input the collected gait video into the trained first video instance segmentation model, to obtain a silhouette sequence of the target person.

[0014] In some embodiments, the key point sequence of the target person is obtained in the following manner:

[0015] obtain a pre-trained second video instance segmentation model trained by using an open source video instance segmentation dataset; wherein the second video instance segmentation model is constructed based on SeqFormer;

[0016] replace segmentation supervision information in the open source video instance segmentation dataset with human key point annotation supervision information;

[0017] fine-tune the pre-trained second video instance segmentation model by using the open source video instance segmentation dataset after the segmentation supervision information is replaced, to obtain a trained second video instance segmentation model;

[0018] input the collected gait video into the trained second video instance segmentation model, to obtain a key point sequence of the target person.

[0019] In some embodiments, the human body part sequence of the target person is obtained in the following manner:

[0020] obtain a pre-trained third video instance segmentation model trained by using an open source video instance segmentation dataset; wherein the third video instance segmentation model is constructed based on SeqFormer;

[0021] after the categories other than the target person in the open source video instance segmentation dataset are all regarded as background, replace them with human body part parsing supervision information;

[0022] fine-tune the pre-trained third video instance segmentation model by using the open source video instance segmentation dataset after the segmentation supervision information is replaced, to obtain a trained third video instance segmentation model;

[0023] input the collected gait video into the trained third video instance segmentation model, to obtain a human body part sequence of the target person.

[0024] In some embodiments, the gait feature of the target person is extracted from the obtained gait sequence, including the following steps:

[0025] identify a category of the gait sequence of the target person;

[0026] According to the identified category of the gait sequence, a corresponding gait recognition model is used to extract gait features from the gait sequence.

[0027] In some embodiments, if the gait sequence is a silhouette sequence, a first gait recognition model is used to extract gait features from the silhouette sequence, wherein the first gait recognition model is constructed based on GaitSet;

[0028] If the gait sequence is a key point sequence, a second gait recognition model is used to extract gait features from the key point sequence, wherein the second gait recognition model is constructed based on GaitGraph;

[0029] If the gait sequence is a human body part sequence, a third gait recognition model is used to extract gait features from the human body part sequence, and the third gait recognition model is constructed based on multi-branch GaitSet.

[0030] In some embodiments, the extracted gait features are compared with a base library to obtain recognition objects that meet a set threshold, including the following steps:

[0031] The extracted gait features are normalized, and the cosine similarity between the gait features and all gait sequences in the base library is calculated respectively;

[0032] According to the calculated cosine similarity, the recognition objects that meet the set threshold are determined, and the determined recognition objects are arranged in order of similarity from high to low.

[0033] In a second aspect, the embodiments of the present application provide a gait recognition device based on video instance segmentation, and the device comprises:

[0034] The acquisition module is configured to simultaneously record the walking state video of the target person by using cameras placed at different angles to obtain walking sequences of the target person at different angles;

[0035] The feature extraction module is configured to extract gait features from the walking sequences of the target person at different angles;

[0036] The recognition module is configured to compare the gait features with a base library to obtain recognition objects that meet a set threshold.

[0037] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a bus. The memory stores machine readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus. The machine readable instructions, when executed by the processor, perform the steps of the gait recognition method based on video instance segmentation of any one of the first aspect.

[0038] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is run by a processor, the steps of the gait recognition method based on video instance segmentation of any one of the first aspect are performed.

[0039] The gait recognition method based on video instance segmentation, the device, the electronic device, and the storage medium provided by the present application can obtain a gait sequence through a trained video instance segmentation model, thereby simplifying the gait recognition steps and improving the model generalization. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0041] Figure 1 A flowchart of the gait recognition method based on video instance segmentation is shown;

[0042] Figure 2 A flowchart of the silhouette sequence of the target person obtained by the embodiments of the present application is shown;

[0043] Figure 3 A structure diagram of the SeqFormer model obtained by the embodiments of the present application is shown;

[0044] Figure 4 A flowchart of the key point sequence of the target person obtained by the embodiments of the present application is shown;

[0045] Figure 5 A flowchart of the body part sequence of the target person obtained by the embodiments of the present application is shown;

[0046] Figure 6 A schematic diagram of a segmentation supervision information visualization result according to an embodiment of the present application is shown.

[0047] Figure 7 A structural schematic diagram of a GaitSet model according to an embodiment of the present application is shown.

[0048] Figure 8 A structural schematic diagram of a GaitGraph model according to an embodiment of the present application is shown.

[0049] Figure 9 A structural schematic diagram of a gait recognition device based on video instance segmentation according to an embodiment of the present application is shown.

[0050] Figure 10 A structural block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0052] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0053] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0054] In view of the technical problems proposed in the background art, the present application provides a gait recognition method and device based on video instance segmentation, an electronic device and a storage medium, which can simplify the gait recognition steps and improve the model generalization.

[0055] Referring to the drawings accompanying the Figure 1 The gait recognition method based on video instance segmentation provided in the embodiments of the present application comprises the following steps:

[0056] S1, a gait video of a target person is collected by a camera, and the gait video is input into a trained video instance segmentation model to obtain a gait sequence of the target person; wherein the gait sequence is a silhouette sequence, a key point sequence or a human body part sequence;

[0057] S2, the gait features of the target person are extracted from the obtained gait sequence; wherein the method for extracting the gait features of the target person is different for the silhouette sequence, the key point sequence or the human body part sequence;

[0058] S3, the extracted gait features are compared with a base library to obtain a recognition object that meets a set threshold.

[0059] The gait recognition method based on video instance segmentation provided in the embodiments of the present application can input the photographed gait video of the target person into the trained video instance segmentation model to directly obtain the gait sequence of the target person, thereby simplifying the gait sequence acquisition step. In the prior art, two steps are often required to obtain the gait sequence, i.e., first detecting the position of the pedestrian, and then obtaining the gait sequence through segmentation / posture estimation / human body analysis, etc. The two-step method has low generalization, different input data need to be designed into different models when the input mode changes, in addition, since the checking model in the first step does not directly act on the acquisition of the corresponding modal gait sequence, it may cause a local optimal solution.

[0060] The gait sequence specifically refers to the contour sequence of the target person when walking, which is used to represent the posture and behavior features of the human body when walking, and can be a series of continuous activities of the human body through the hip, knee, ankle and toes. In this embodiment, the gait sequence can be a silhouette sequence, a key point sequence or a human body part sequence, which is output by different trained video instance segmentation models.

[0061] Specifically, in an embodiment, referring to the drawings accompanying the Figure 2 The silhouette sequence of the target person is obtained by the following method:

[0062] S101, a pre-trained first video instance segmentation model trained by using an open source video instance segmentation dataset is obtained; wherein the first video instance segmentation model is constructed based on SeqFormer;

[0063] S102, replace the segmentation supervision information in the open source video instance segmentation dataset to regard all categories except the target personnel as background;

[0064] S103, fine-tune the pre-trained first video instance segmentation model using the open source video instance segmentation dataset after replacing the segmentation supervision information, to obtain a trained first video instance segmentation model;

[0065] S104, input the collected gait video into the trained first video instance segmentation model to obtain a silhouette sequence of the target personnel.

[0066] In steps S101-S104, the open source video instance segmentation dataset is derived from the YouTube-VIS 2019 dataset and the COCO dataset, which is used as a first training sample set to train a pre-constructed first video instance segmentation model to obtain a pre-trained first video instance segmentation model. In this embodiment, the pre-trained first video instance segmentation model is constructed based on SeqFormer. The structural diagram of the SeqFormer model can be seen in the accompanying drawings Figure 3 . The SeqFormer model is divided into three parts: a CNN backbone network, a Transformer encoder, and a query decomposition encoder. Given an input frame , the CNN backbone network independently extracts the feature map of each frame; then a 1*1 convolution layer is added to reduce the channel dimension to 256, generating a feature map , which is input to the Transformer encoder after adding position encoding. The output feature has the same dimension as the input feature; then the encoded feature is input to the query decomposition Transformer decoder. First, a fixed-dimensional learnable instance query vector is defined. In the first decoder layer, the initial instance query is shared to each frame to do deformable attention , where is the boundary box query from the first layer decoder on the t-th frame, is the instance query, is the feature on the t-th frame; in the l-th layer (l>1) after that, the query of the previous layer is used as input ; after passing through all the decoders, an instance query and T boundary box queries are obtained for each instance. The instance query is a shared video-level instance representation, and the boundary box query contains the position information for predicting the boundary box on each frame. Finally, there are three output heads respectively outputting instance masks (mask head), instance categories (category head), and instance bounding boxes (bounding box head).

[0067] In addition, after the pre-trained first video instance segmentation model is trained by the first training sample set (extracted from the YouTube-VIS 2019 dataset and the COCO dataset), the pre-trained first video instance segmentation model needs to be fine-tuned. Specifically, the categories in the segmentation supervision information in the first training sample set except pedestrians are regarded as background, and the visualization diagrams before and after the segmentation supervision information is replaced can be seen from the accompanying drawings of the specification Figure 6 , where (a) is the visualization diagram of the original segmentation supervision information, and (b) is the visualization diagram of the segmentation supervision information in which the categories except pedestrians are regarded as background. Then, the pre-trained first video instance segmentation model is fine-tuned on the first training sample set with replaced supervision information, and a trained first video instance segmentation model is obtained, so that the collected gait video is input into the trained first video instance segmentation model to obtain a silhouette sequence of the target person.

[0068] The process of training the first video instance segmentation model and fine-tuning is a technical means known to those skilled in the art, and will not be repeated here.

[0069] Referring to the accompanying drawings of the specification Figure 4 , the key point sequence of the target person is obtained by the following method:

[0070] P101, obtaining a pre-trained second video instance segmentation model trained by using an open-source video instance segmentation dataset; wherein the second video instance segmentation model is constructed based on SeqFormer;

[0071] P102, replacing the segmentation supervision information in the open-source video instance segmentation dataset with human key point annotation supervision information;

[0072] P103, fine-tuning the pre-trained second video instance segmentation model by using the open-source video instance segmentation dataset with replaced segmentation supervision information to obtain a trained second video instance segmentation model;

[0073] P104, inputting the collected gait video into the trained second video instance segmentation model to obtain a key point sequence of the target person.

[0074] In steps S101-S104, a second video instance segmentation model constructed based on SeqFormer is first trained by a second training sample set (extracted from the YouTube-VIS 2019 dataset and the COCO dataset) to obtain a pre-trained first video instance segmentation model, and then the parameters of the pre-trained first video instance segmentation model are fine-tuned by replacing the second training sample set with segmentation supervision information, so as to finally obtain a trained second video instance segmentation model. Thus, the collected gait video is input into the trained second video instance segmentation model to obtain a key point sequence of the target person. The schematic diagrams before and after the replacement of the segmentation supervision information can be seen in FIGS. 1A-1C of the specification. Figure 6 , wherein (a) is a visualized schematic diagram of the original segmentation supervision information, and (c) is a visualized schematic diagram of the segmentation supervision information replaced by human key point annotation supervision information.

[0075] Referring to FIGS. 2A-2C of the specification, Figure 5 The human part sequence of the target person is obtained by the following method:

[0076] Q101, obtaining a pre-trained third video instance segmentation model trained by using an open-source video instance segmentation dataset; wherein the third video instance segmentation model is constructed based on SeqFormer;

[0077] Q102, after regarding all categories other than the target person in the open-source video instance segmentation dataset as background, replacing the segmentation supervision information with human part analysis supervision information;

[0078] Q103, fine-tuning the pre-trained third video instance segmentation model by using the open-source video instance segmentation dataset with the replaced segmentation supervision information to obtain a trained third video instance segmentation model;

[0079] Q104, inputting the collected gait video into the trained third video instance segmentation model to obtain a human part sequence of the target person.

[0080] In steps Q101-Q104, a third video instance segmentation model constructed based on SeqFormer is first trained by a third training sample set (extracted from the YouTube-VIS 2019 dataset and the COCO dataset) to obtain a pre-trained third video instance segmentation model, and then the parameters of the pre-trained third video instance segmentation model are fine-tuned by replacing the third training sample set with segmentation supervision information, so as to finally obtain a trained third video instance segmentation model. Thus, the collected gait video is input into the trained third video instance segmentation model to obtain a key point sequence of the target person. The schematic diagrams before and after the replacement of the segmentation supervision information can be seen in FIGS. 1A-1C of the specification. Figure 6, Fig. (a) is a visual diagram of the original segmentation supervision information, and Fig. (d) is a visual diagram of replacing the categories other than pedestrians as background and replacing them with human body part analysis supervision information.

[0081] In step S2, the gait feature is extracted from the gait sequence according to the category of the identified target personnel gait sequence. When the silhouette sequence is input into the gait recognition model, the first gait recognition model is used to extract the gait feature from the silhouette sequence, and the first gait recognition model is constructed based on GaitSet. The structure of the GaitSet model can be referred to in the description accompanying drawings Figure 7 , including CNN, set pooling layer (Setpooling) and horizontal pyramid pooling layer, wherein the silhouette sequence is first input into the CNN to extract frame silhouette features, the frame-based features are fused into set-based features through the set pooling layer (Setpooling), and then the frame-level features and set-level features are mapped through the horizontal pyramid pooling layer to obtain the gait feature finally.

[0082] When the key point sequence is input into the gait recognition model, the second gait recognition model is used to extract the gait feature from the key point sequence, and the second gait recognition model is constructed based on GaitGrapht. The structure of the GaitGraph model can be referred to in the description accompanying drawings Figure 8 , including ResGCN module and full connection layer, wherein the key point sequence is first input into a plurality of ResGCN modules, wherein ResGCN includes graph convolution layer, 2d convolution layer, bottleneck layer, etc., then average pooling is performed on the output features, and the full connection layer is input to output the final gait feature.

[0083] When the human body part sequence is input into the gait recognition model, the third gait recognition model is used to extract the gait feature from the human body part sequence, and the third gait recognition model is constructed based on the multi-branch GaitSet. Wherein, the silhouette sequence of K parts is input into the GaitSet to extract K groups of features, and then the most discriminative part of different dimensions in the K groups of part features is selected to combine into the final gait feature.

[0084] In step S3, the gait feature is first normalized, and the cosine similarity between all gait sequences in the base library is calculated. Then, according to the calculated cosine similarity, the recognition object satisfying the set threshold is determined, and the determined recognition object is arranged in order from high to low according to the similarity. The base library is composed of a plurality of candidate walking sequence groups determined by various gait data captured in real scenes. In this embodiment, the gait feature is used as the retrieval object, and the base library is represented as , and a single gait sequence in the base library is represented as . For the dimension of gait features, the calculation formula is as follows:

[0085]

[0086] Then, the above formula is used to calculate the cosine similarity with all gait sequences in the base library, and then the query results meeting a certain threshold are returned and arranged in descending order of similarity. For example, if the threshold is set to 0.7, the cosine similarity of all gait sequences in the base library is arranged in descending order.

[0087] It can be seen that the gait recognition method based on video instance segmentation provided in the application can input the gait video of the target person captured into the trained video instance segmentation model to obtain the gait sequence of the target person, and then input the gait sequence into the gait recognition model to extract gait features, thereby simplifying the acquisition step of the gait sequence, and using the video instance segmentation model only needs to be fine-tuned according to the sequence category, which has high generalization and ensures that the optimization target of the detection task is consistent with the corresponding modal gait sequence acquisition task.

[0088] Based on the same inventive concept, the application also provides a gait recognition device based on video instance segmentation. Since the principle of the device in the application solves the problem similarly to the above-mentioned gait recognition method based on video instance segmentation, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0089] As shown in the accompanying drawings, Figure 9 the application also provides a gait recognition device based on video instance segmentation. The device comprises:

[0090] The acquisition module 901 is configured to acquire the gait video of the target person through a camera and input the gait video into a trained video instance segmentation model to obtain the gait sequence of the target person. The gait sequence is a silhouette sequence, a key point sequence or a human body part sequence.

[0091] The extraction module 902 is configured to extract the gait features of the target person from the obtained gait sequence. The method for extracting the gait features of the target person is different for the silhouette sequence, the key point sequence or the human body part sequence.

[0092] The recognition module 903 is configured to compare the extracted gait features with a base library to obtain a recognition object meeting a set threshold.

[0093] In some embodiments, the acquisition module 901 obtains the silhouette sequence of the target person, comprising:

[0094] obtain a pre-trained first video instance segmentation model trained by using an open-source video instance segmentation dataset; wherein the first video instance segmentation model is constructed based on SeqFormer;

[0095] replace segmentation supervision information in the open-source video instance segmentation dataset so as to regard all categories other than the target person as background;

[0096] fine-tune the pre-trained first video instance segmentation model by using the open-source video instance segmentation dataset after the segmentation supervision information is replaced, to obtain a trained first video instance segmentation model;

[0097] input the collected gait video into the trained first video instance segmentation model, to obtain a silhouette sequence of the target person.

[0098] In some embodiments, the acquisition module 901 obtains a key point sequence of the target person, comprising:

[0099] obtain a pre-trained second video instance segmentation model trained by using an open-source video instance segmentation dataset; wherein the second video instance segmentation model is constructed based on SeqFormer;

[0100] replace segmentation supervision information in the open-source video instance segmentation dataset with human key point annotation supervision information;

[0101] fine-tune the pre-trained second video instance segmentation model by using the open-source video instance segmentation dataset after the segmentation supervision information is replaced, to obtain a trained second video instance segmentation model;

[0102] input the collected gait video into the trained second video instance segmentation model, to obtain a key point sequence of the target person.

[0103] In some embodiments, the acquisition module 901 obtains a human body part sequence of the target person, comprising:

[0104] obtain a pre-trained third video instance segmentation model trained by using an open-source video instance segmentation dataset; wherein the third video instance segmentation model is constructed based on SeqFormer;

[0105] replace, after regarding all categories other than the target person in the open-source video instance segmentation dataset as background, human body part analysis supervision information;

[0106] The pre-trained third video instance segmentation model is fine-tuned by using the open source video instance segmentation dataset after the replacement of the segmentation supervision information, to obtain a trained third video instance segmentation model.

[0107] The collected gait video is input into the trained third video instance segmentation model, to obtain a human body part sequence of the target person.

[0108] In some embodiments, the extraction module 902 extracts gait features of the target person from the obtained gait sequence, including:

[0109] A category of the obtained gait sequence of the target person is identified.

[0110] According to the identified category of the gait sequence, a corresponding gait recognition model is used to extract gait features from the gait sequence; if the gait sequence is a silhouette sequence, a first gait recognition model is used to extract gait features from the silhouette sequence, wherein the first gait recognition model is constructed based on GaitSet; if the gait sequence is a key point sequence, a second gait recognition model is used to extract gait features from the key point sequence, wherein the second gait recognition model is constructed based on GaitGraph; if the gait sequence is a human body part sequence, a third gait recognition model is used to extract gait features from the human body part sequence, wherein the third gait recognition model is constructed based on multi-branch GaitSet.

[0111] In some embodiments, the identification module 903 compares the extracted gait features with a base library to obtain identification objects that meet a set threshold, including:

[0112] The extracted gait features are normalized, and the cosine similarity between the gait features and all gait sequences in the base library is calculated respectively.

[0113] According to the calculated cosine similarity, identification objects that meet a set threshold are determined, and the determined identification objects are arranged in order of similarity from high to low.

[0114] The application provides a gait recognition device based on video instance segmentation. A gait video of a target person is input into a trained video instance segmentation model through a collection module to obtain a gait sequence of the target person; wherein the gait sequence is a silhouette sequence, a key point sequence or a human body part sequence; a gait feature of the target person is extracted from the obtained gait sequence through an extraction module; wherein the method for extracting the gait feature of the target person is different for the silhouette sequence, the key point sequence or the human body part sequence; the extracted gait feature is compared with a database through an identification module to obtain an identification object meeting a set threshold. Thus, the gait recognition steps are simplified, and the model generalization is improved.

[0115] Based on the same concept of the application, the specification is attached Figure 10 As shown in the accompanying drawings, the electronic device 1000 provided by the embodiments of the application includes at least one processor 1001, at least one network interface 1004 or other user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the components. The electronic device 1000 can optionally include a user interface 1003, including a display (for example, a touch screen, an LCD, a CRT, holographic imaging (Holographic) or a projector (Projector), etc.), a keyboard or a clicking device (for example, a mouse, a trackball, a touchpad or a touch screen, etc.).

[0116] The memory 1005 can include read-only memory and random access memory, and provide instructions and data for the processor 1001. A part of the memory 1005 can also include a non-volatile random access memory (NVRAM).

[0117] In some embodiments, the memory 1005 stores the following elements, protected modules or data structures, or their subsets, or their extended sets:

[0118] An operating system 10051 includes various system programs, used to realize various basic services and process hardware-based tasks;

[0119] An application program module 10052 includes various application programs, such as a desktop (launcher), a media player (Media Player), a browser (Browser), etc., used to realize various application services.

[0120] In the embodiments of the application, the processor 1001 is used to execute the steps in a gait recognition method based on video instance segmentation by calling the programs or instructions stored in the memory 1005, which can simplify the gait recognition steps and improve the model generalization.

[0121] The application further provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, steps in the gait recognition method based on video instance segmentation are executed.

[0122] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the gait recognition method based on video instance segmentation can be executed.

[0123] In the embodiments provided in the application, it should be understood that the disclosed apparatus and method can be implemented by other manners. The apparatus embodiments described above are only schematic, for example, the division of units is only a logical function division, and another division manner can be used in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, and can be electrical, mechanical or other forms.

[0124] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0125] In addition, each functional unit in the embodiments provided in the application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0126] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0127] Finally, it should be noted that the above examples are merely specific embodiments of the present application, and are used to illustrate the technical solutions of the present application, but are not a limitation thereof, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing examples, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application, and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application. All should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A gait recognition method based on video instance segmentation, characterized in that, The method includes the following steps: The gait video of the target person is captured by a camera, and the gait video is input into a trained video instance segmentation model to obtain the gait sequence of the target person; wherein, the gait sequence is a silhouette sequence, a key point sequence, and a body part sequence; Gait features of the target person are extracted from the obtained gait sequence. Different methods are used for extracting the gait features of the target person for the silhouette sequence, the keypoint sequence, and the body part sequence. The silhouette sequence of the target person is obtained as follows: a pre-trained first video instance segmentation model trained using an open-source video instance segmentation dataset is acquired; the first video instance segmentation model is built based on SeqFormer. The segmentation supervision information in the open-source video instance segmentation dataset is replaced to treat all categories other than the target person as background. The pre-trained first video instance segmentation model is fine-tuned using the open-source video instance segmentation dataset with the replaced segmentation supervision information to obtain a trained first video instance segmentation model. The collected gait video is input into the trained first video instance segmentation model to obtain the silhouette sequence of the target person. The keypoint sequence of the target person is obtained as follows: a pre-trained second video instance segmentation model trained using an open-source video instance segmentation dataset is acquired; the second video instance segmentation model is based on SeqFormer. The process involves: constructing a pre-trained second video instance segmentation model; replacing the segmentation supervision information in the open-source video instance segmentation dataset with human keypoint annotation supervision information; fine-tuning the pre-trained second video instance segmentation model using the open-source video instance segmentation dataset with the replaced segmentation supervision information to obtain a trained second video instance segmentation model; inputting the collected gait video into the trained second video instance segmentation model to obtain the keypoint sequence of the target person; and obtaining the human body part sequence of the target person through the following method: acquiring a pre-trained third video instance segmentation model trained using the open-source video instance segmentation dataset; wherein the third video instance segmentation model is constructed based on SeqFormer; treating all categories other than the target person in the open-source video instance segmentation dataset as background and replacing them with human body part parsing supervision information; fine-tuning the pre-trained third video instance segmentation model using the open-source video instance segmentation dataset with the replaced segmentation supervision information to obtain a trained third video instance segmentation model; and inputting the collected gait video into the trained third video instance segmentation model to obtain the human body part sequence of the target person. The extracted gait features are compared with the baseline database to obtain the recognition objects that meet the set threshold.

2. The gait recognition method based on video instance segmentation according to claim 1, characterized in that, Extracting the gait features of the target person from the obtained gait sequence includes the following steps: Identify the category of the obtained gait sequence of the target person; Based on the identified gait sequence category, the corresponding gait recognition model is used to extract gait features from the gait sequence.

3. The gait recognition method based on video instance segmentation according to claim 2, characterized in that, If the gait sequence is a silhouette sequence, gait features are extracted from the silhouette sequence using a first gait recognition model, wherein the first gait recognition model is constructed based on GaitSet; If the gait sequence is a keypoint sequence, gait features are extracted from the keypoint sequence using a second gait recognition model, wherein the second gait recognition model is constructed based on GaitGraph; If the gait sequence is a human body part sequence, gait features are extracted from the human body part sequence using a third gait recognition model, which is constructed based on a multi-branch GaitSet.

4. The gait recognition method based on video instance segmentation according to claim 3, characterized in that, The step of comparing the extracted gait features with a baseline database to obtain recognition objects that meet a set threshold includes the following steps: The extracted gait features are normalized, and the cosine similarity between them and all gait sequences in the base database is calculated. Based on the calculated cosine similarity, the identification objects that meet the set threshold are determined, and the identified identification objects are arranged in descending order of similarity.

5. A gait recognition device based on video instance segmentation, characterized in that, The device includes: The acquisition module is used to acquire gait videos of the target person through a camera, and input the gait videos into a trained video instance segmentation model to obtain the gait sequence of the target person; wherein, the gait sequence is a silhouette sequence, a key point sequence, and a human body part sequence; An extraction module is used to extract gait features of the target person from the obtained gait sequence. Different methods are used for extracting the gait features of the target person for the silhouette sequence, the keypoint sequence, and the body part sequence. Specifically, the silhouette sequence of the target person is obtained as follows: A pre-trained first video instance segmentation model trained using an open-source video instance segmentation dataset is acquired; the first video instance segmentation model is built based on SeqFormer. The segmentation supervision information in the open-source video instance segmentation dataset is replaced to treat all categories other than the target person as background. The pre-trained first video instance segmentation model is fine-tuned using the open-source video instance segmentation dataset with the replaced segmentation supervision information to obtain a trained first video instance segmentation model. The collected gait video is input into the trained first video instance segmentation model to obtain the silhouette sequence of the target person. The keypoint sequence of the target person is obtained as follows: A pre-trained second video instance segmentation model trained using an open-source video instance segmentation dataset is acquired; the second video instance segmentation model is based on SeqFormer. The process involves: constructing a SeqFormer model; replacing the segmentation supervision information in the open-source video instance segmentation dataset with human keypoint annotation supervision information; fine-tuning the pre-trained second video instance segmentation model using the open-source video instance segmentation dataset with the replaced segmentation supervision information to obtain a trained second video instance segmentation model; inputting the collected gait video into the trained second video instance segmentation model to obtain the keypoint sequence of the target person; and obtaining the human body part sequence of the target person in the following manner: acquiring a pre-trained third video instance segmentation model trained using the open-source video instance segmentation dataset; wherein the third video instance segmentation model is constructed based on SeqFormer; treating all categories other than the target person in the open-source video instance segmentation dataset as background and replacing them with human body part parsing supervision information; fine-tuning the pre-trained third video instance segmentation model using the open-source video instance segmentation dataset with the replaced segmentation supervision information to obtain a trained third video instance segmentation model; and inputting the collected gait video into the trained third video instance segmentation model to obtain the human body part sequence of the target person. The recognition module is used to compare the extracted gait features with the baseline database to obtain recognition objects that meet the set threshold.

6. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the gait recognition method based on video instance segmentation as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the gait recognition method based on video instance segmentation as described in any one of claims 1 to 4.

Images