A method for re-identification of people under care outdoors based on multiple fusion features

Abstract

The present invention is applicable to the field of artificial intelligence, and provides a method for re-identifying people outside the care center based on multiple fusion features, including: extracting RGB image sequence features of people inside the care center, and aggregating them into appearance features of the people through a temporal attention mechanism; extracting gait energy features of people inside the care center, and identifying walking state features; performing perspective estimation on people outside the care center, calculating the importance weights of appearance features, gait energy features, and walking state features of the people, and establishing a fusion model with importance weights; performing target classification detection on outdoor people, realizing image search of people outside the care center, and comparing the search results with the recognition structure obtained by the fusion model to complete the re-identification of people outside the care center. The present invention integrates the appearance features, gait energy features, and walking state features of the people, designs a fusion model, realizes re-identification of people, and improves the accuracy of image recognition.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and intelligent optimization computing technology, and in particular relates to a method for re-identifying a person under care outdoors based on multiple fusion features. Background Art

[0002] The person under care is a person who needs care, such as a patient who needs care. Through care management, when the patient has an accident, it can be discovered in the first time, which improves safety. In the context of Digital China, the management of the person under care has also moved towards digitalization. With the help of information technologies such as the Internet of Things, cloud computing, big data, and artificial intelligence, multi-dimensional data can be obtained and analyzed for fusion features, providing a feasible means for the intelligent management of the person under care. Among them, the re-identification of the person under care outdoors is one of the important intelligent management issues. This involves a core issue, that is, the difficulty in collecting image information of the person under care outdoors, which causes the low re-identification accuracy.

[0003] The difficulty in collecting image information of outdoor persons under care is caused by the installation requirements of equipment such as cameras. This is a practical requirement for safety work and is difficult to change. In order to solve the problem of outdoor re-identification of persons under care, the existing technology mostly adopts single feature recognition methods such as image recognition. Therefore, in addition to image correction to improve image recognition accuracy, it is urgent to integrate features other than image feature recognition. At present, the more prominent feature is gait energy feature recognition. In addition, in the management of persons under care, sensor data in the bracelets in the rooms under care can also be obtained, such as extracting acceleration sensor data to identify the walking status of people. Fusion of the above features and calculation of the reasonable weight of each feature can significantly improve the accuracy of feature recognition. However, the identification of reasonable weights needs to be flexibly handled according to the actual scenario and the selection of fusion features.

[0004] In the actual outdoor personnel re-identification scenario, outdoor personnel are mainly divided into medical staff, staff and personnel being cared for, and their appearance features such as clothing are clearly distinguishable. Therefore, to improve the re-identification accuracy of outdoor personnel being cared for, we can also start with the personnel type, and further distinguish the personnel type to reduce the number of people that need to be identified to improve the accuracy. In addition to this special background, general technical solutions rarely consider both deep learning recognition technology and personnel re-identification calculation process to further improve image recognition accuracy: choosing a deep recognition method with excellent performance, choosing a suitable distance measurement and sorting method, and adopting a good buffering and updating mechanism can effectively improve the re-identification accuracy of outdoor personnel being cared for. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a method for re-identifying outdoor persons under care based on multiple fusion features, aiming to solve the technical problem that the existing re-identification model for outdoor persons under care has insufficient recognition accuracy.

[0006] The present invention adopts the following technical solution:

[0007] The method for re-identifying a person outside a cared area based on multiple fusion features comprises the following steps:

[0008] Step S1, extracting the RGB image sequence features of the people in the guarded room, aggregating them into the appearance features of the people through the temporal attention mechanism, and performing perspective transformation on the RGB images of the people outside the guarded room;

[0009] Step S2, extracting the gait energy characteristics of the person in the care room, and identifying the walking state characteristics in combination with the acceleration sensor worn by the person;

[0010] Step S3, estimate the viewing angle of the person outside the guarded room, calculate the importance weights of the person's appearance features, gait energy features, and walking state features, and establish a fusion model with importance weights;

[0011] Step S4: perform target classification detection on outdoor nursing staff, outdoor staff and outdoor persons being cared for, realize image search of outdoor persons being cared for, compare the search results with the recognition structure obtained by the fusion model, and complete the re-identification of outdoor persons being cared for.

[0012] The beneficial effects of the present invention are as follows: the present invention provides a more reasonable re-identification method based on the fusion of personnel appearance features, gait energy features and walking state features, and reasonably calculates the importance weights of the three features, designs a fusion model, and then realizes personnel re-identification through image target classification detection and image search, to further improve the image recognition accuracy; the scheme of the present invention can provide a new high-precision optional scheme for the detection of outdoor cared persons. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The invention provides a method for re-identifying a person outside a cared area based on multiple fusion features. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0015] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0016] like Figure 1 As shown, the method for re-identifying a person outside a cared area based on multiple fusion features provided in this embodiment includes the following steps:

[0017] Step S1: extract the RGB image sequence features of the people in the guarded room, aggregate them into the appearance features of the people through the temporal attention mechanism, and perform perspective transformation on the RGB images of the people outside the guarded room.

[0018] This step extracts the appearance features of people outside the guarded room. Specifically, the RGB image sequence features of people inside the guarded room are used as the training set and aggregated into appearance features through temporal attention. In addition, the RGB images of people outside the guarded room are also extracted for perspective transformation preprocessing as the test set after the subsequent model training is completed. The specific process of this step is as follows:

[0019] S11. Extract the RGB image sequence features of the people in the room being cared for, and then aggregate them into the appearance features of the people through the temporal attention mechanism.

[0020] Based on deep learning pedestrian re-identification, the RGB images of people in the guarded room are first used as the input training set. The general backbone network uses ResNet to extract features, and the image is obtained by processing video data based on temporal attention.

[0021] In this embodiment, the appearance feature expression of the person extracted from the person in the care room based on time attention is as follows:

[0022]

[0023] Among them, Q a is the aggregated person appearance feature, n is the number of RGB images of the person in the room being cared for, A t ∈P H×W×L ∪C H×W×L is the t-th input RGB image with spatial features P and temporal features C, H×W×L represents the feature dimension after ResNet extraction, and f a (A t ) represents the spatial domain ResNet attention network, g a (A t ) represents the time domain ResNet attention network.

[0024] S12, obtaining an RGB image of the person being cared for outside the room, and performing perspective transformation.

[0025] Since the cameras outside the guarded room are located at a high position, it is impossible to obtain a straight-on image of the person. Through perspective transformation, an image with higher recognition quality can be obtained. The specific transformation formula is as follows:

[0026]

[0027] The perspective transformation matrix is: The points to be moved, i.e. the source and target points, are: Fixed point, that is, the target point to move to is:

[0028] Transform from two-dimensional space to three-dimensional space. Since the image is in a two-dimensional plane, divide it by Z, and (X~, Y~, Z~) represents a point on the image.

[0029] In addition 33 =1, and substituting it into the formula after dividing by Z, we get:

[0030]

[0031] Substituting the four points, we can get 8 equations, and then we can solve the perspective transformation matrix A. Perspective transformation is a public technology and will not be described here.

[0032] Step S2: extract the gait energy characteristics of the person in the care room, and identify the walking state characteristics in combination with the acceleration sensor worn by the person.

[0033] The specific process of this step is as follows:

[0034] S21. Use the ResNet attention network to extract the gait energy characteristics of people in the care room.

[0035] First, we select the gait database CASIA-B (published by the State Key Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences in 2005), which includes 124 people. We select 6 normal walking (NM) sequences (NM-01 to NM-06) for each person, and the 11 viewing angles are 0° to 180°. A camera is set up at an interval of 18°, and the 11 cameras shoot at the same time, so there are 11 angles of 0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 144°, 162°, and 180°. The pedestrian profile data is a binary image obtained by decomposing the video of each pedestrian into several frames using the traditional segmentation algorithm. According to the requirements of the above database, the same specification data can also be obtained for the indoor personnel and participate in the subsequent mixed training.

[0036] The gait energy feature is also extracted using the ResNet attention network. The expression for gait energy feature extraction is:

[0037] Q b =fb(R)

[0038] where R∈P H×W×L is the input gait energy map image, f b (R) represents the gait energy features of the gait energy map image extracted by the ResNett attention network.

[0039] S22. According to the three-axis vector data of the acceleration sensor, a standard walking state library of personnel is established, and the standard deviation of vertical axis change is established as a feature quantity. The cared person is used as an identification category to obtain the walking state characteristics of the person.

[0040] The people in the care room are wearing wearable acceleration sensors, which can obtain some parameters of the people's actions. This step establishes a standard state library of the walking state of the detained people based on the three-axis vector data of the acceleration sensor, establishes the standard deviation of the vertical axis change as the feature quantity, and the cared person is the identification category.

[0041] Step S3: estimate the viewing angle of the person being cared for outside the room, calculate the importance weights of the person's appearance features, gait energy features, and walking state features, and establish a fusion model with importance weights.

[0042] Since the images of people indoors are labeled with angles, the images of people outdoors need to estimate the angles based on deep learning prediction. The fusion model of the present invention integrates appearance features, gait energy features, and walking state features, and expresses the fusion model through three importance weights. The specific process of this step is as follows:

[0043] S31. Classify and identify the perspective-transformed RGB image of the outdoor person being cared for according to the set classification angle, that is, view angle estimation.

[0044] For the images of people in the cared room to be estimated, classification and recognition are performed at a total of 11 angles, including 0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 144°, 162°, and 180°. Deep learning classification algorithms or traditional machine learning algorithms can be used. The relevant classification learning algorithms are relatively common and will not be described in detail here.

[0045] S32. Calculate the importance weights of the person's appearance features and gait energy features.

[0046] Let x be the estimated result of the perspective of the person outside the guarded area. Since the gait feature is most obvious at 90° and then weakens at 0° and 180°, the weight of the appearance feature at 90° is designed as a and gait feature weight ω b Each is 0.5, 0° and 180° to weaken the gait characteristics in turn, and the following weight calculation formula can be obtained:

[0047]

[0048] Among them, ω a is the importance weight of the appearance feature, ωb is the importance weight of the gait energy feature, and x is the estimated result of the perspective of the person being cared for outside the room.

[0049] S33. Establish a fusion model with importance weights, and then use the fusion model to perform target detection training for outdoor personnel under care.

[0050] In practice, when it is close to 0°, the person is close to frontal view, and the formula performs better at this time; when it is close to 180°, the person is close to back view, and the appearance characteristics and gait energy characteristics are not obvious at this time. This embodiment selects 135° as the dividing point. When the estimated angle is >135°, the standard deviation of the vertical axis change of the person being cared for is identified through continuous video images, and the specific person being cared for is directly identified through classification algorithms such as SVM (support vector machine).

[0051] The entire fusion model can be classified as follows:

[0052]

[0053] where ω c is the importance weight of the walking state feature. When x>135°, ω c =1, otherwise ω c =0,f cd (·) represents the encoder in the autoencoder, f c (M) is the walking state feature.

[0054] It can be seen from the above expression that when the angle is less than or equal to 135°, the second part of the model expression is 0, that is, the fusion model is only related to the appearance features and gait energy features, and the expression is This means that the autoencoder is used to fuse the appearance features extracted from the RGB image and the gait energy features extracted from the gait energy map. When the angle is greater than 135°, the first part of the model expression is 0, that is, the fusion model is only related to the walking state features, and can be identified by classification algorithms such as SVM based on the standard deviation of the vertical axis change.

[0055] Therefore, the above fusion model expression combines the two perspectives and adopts the corresponding fusion model expression.

[0056] The above fusion model can be tested with the aforementioned perspective transformed RGB image of the person outside the care center. After testing, the recognition accuracy of this embodiment is very high.

[0057] Step S4: perform target classification detection on outdoor nursing staff, outdoor staff and outdoor persons being cared for, realize image search of outdoor persons being cared for, compare the search results with the recognition structure obtained by the fusion model, and complete the re-identification of outdoor persons being cared for.

[0058] The above has completed the establishment, training and testing of the fusion model. Re-identification mainly involves target detection and image search and comparison. For outdoor personnel images, after processing through the above fusion model, the recognition results of the outdoor personnel being cared for can be obtained. This step uses image search and recognition results for comparison to ensure accuracy.

[0059] The specific process of this step is as follows:

[0060] S41. A bottom-up approach based on semantic segmentation is used to classify and detect targets of outdoor caregivers, outdoor staff, and outdoor personnel being cared for in the current outdoor image.

[0061] For target classification detection of outdoor caregivers, staff and outdoor personnel being cared for, a bottom-up approach based on semantic segmentation can be adopted as follows: First, pixel-by-pixel classification is performed through semantic segmentation to extract the personnel images, and then clustering or other metric learning methods are used to distinguish different instances of the same type. Take the clustering method of the discriminant loss function as an example:

[0062] First, the discriminant loss function performance at each pixel is predicted, so that the vector distance of the belonging person is closer and the vector distance of the non-belonging person is farther; then the mean-shift clustering is used for post-processing, so that a vector can be solved so that the center of the circle moves in the direction with the highest density of the data set. Each iteration calculates the synthetic vector of the force and moves the center of the circle to a new equilibrium position along the synthetic direction of the force.

[0063] S42, then perform image search on the images classified as outdoor personnel under care using a K-nearest neighbor algorithm, and compare the search results with the recognition structure obtained by the fusion model.

[0064] The K nearest neighbor algorithm process is:

[0065] 421. Obtain K nearest neighbor samples of the image feature vector: The image features of the image of the person under care outside the classified area are i-segment query vector ρ, and the distance from each sub-segment of the h residual vectors in the cluster where the query vector ρ is located to the corresponding sub-segment of the query vector is calculated to form a distance set D of h distances. The first distance set in the query vector ρ is:

[0066] D={D i 1,1 |1≤i≤h}

[0067] Sort the distances in the distance set from small to large, and get a total of h ranking numbers from 1 to h. Label the sub-segments corresponding to the distances with the ranking numbers. Each sub-segment of each residual vector has a label, and get a set of codes encoded with h ranking numbers.

[0068] In the encoding set, the sum of the numbers encoded for each number is calculated, and the values ​​are sorted from small to large to form a new set. The samples corresponding to the first K values ​​in the new set are the K nearest neighbor samples of the query vector.

[0069] 422. Establish a buffer library: by querying the K nearest neighbor samples of the vector, the image target of the outdoor caregiver to be inspected in each frame is first matched with the library of indoor caregivers. Since the target to be inspected does belong to a certain ID in the library, the match may fail. In addition, since the target detection may have false detection, assigning a new ID at this time will cause wrong matching and false detection. Therefore, the matching result is first put into the buffer library, and then the target to be inspected that failed to match will be matched with the buffer library. If the match is successful, the feature is added to the buffer library. When the number of feature vectors under the corresponding ID in the buffer library meets the limit, the target to be inspected is added to the library and assigned a new ID; if the match still fails, a new ID is created in the buffer library and the feature vector of the target to be inspected is stored. The IDs in the buffer library that failed to match for several consecutive frames will be deleted.

[0070] 423. Dynamic feature update. Calculate the ratio of the intersection area and the union area of ​​the two rectangles between the target frames of the successfully matched target to be inspected. If the value is less than the threshold, it is considered that there is no large overlap between the two rectangles. Store the image features of the target to be inspected in the library under the corresponding ID. If the number of features of the ID meets the limit, randomly replace the non-initial image features.

[0071] In this embodiment, target classification detection and K nearest neighbor algorithm for image search are existing methods, which will not be described in detail here. The searched images are compared with the recognition images of the fusion model to improve the recognition accuracy.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for re-identifying people outside the care center based on multiple fusion features, characterized in that: The method comprises the following steps: Step S1, extracting the RGB image sequence features of the people in the guarded room, aggregating them into the appearance features of the people through the temporal attention mechanism, and performing perspective transformation on the RGB images of the people outside the guarded room; Step S2, extracting the gait energy characteristics of the person in the care room, and identifying the walking state characteristics in combination with the acceleration sensor worn by the person; Step S3, estimate the viewing angle of the person outside the guarded room, calculate the importance weights of the person's appearance features, gait energy features, and walking state features, and establish a fusion model with importance weights; Step S4: perform target classification detection on outdoor nursing staff, outdoor staff and outdoor persons being cared for, realize image search of outdoor persons being cared for, compare the search results with the recognition structure obtained by the fusion model, and complete re-identification of outdoor persons being cared for; The specific process of step S1 is as follows: S11. Extract the RGB image sequence features of the people in the room being cared for, and then aggregate them into the appearance features of the people through the temporal attention mechanism. The expression of the appearance features of the people is as follows: Among them, Q a is the aggregated person appearance feature, n is the number of RGB images of people in the room being cared for, A t ∈P H×W×L ∪C H ×W×L is the t-th input RGB image with spatial features P and temporal features C, H×W×L represents the feature dimension after ResNet extraction, and f a (A t ) represents the spatial domain ResNet attention network, g a (A t ) represents the time domain ResNet attention network; S12, obtaining an RGB image of the person being cared for outside the room, and performing perspective transformation; The specific process of step S2 is as follows: S21. Use the ResNet attention network to extract the gait energy characteristics of the people in the care room: Q b =f b (R) where R∈P H×W×L is the input gait energy map image, f b (R) represents the gait energy features of the gait energy map image extracted by the ResNett attention network; S22, according to the three-axis vector data of the acceleration sensor, establish a standard state library of walking of personnel, and establish the standard deviation of vertical axis change as a feature quantity, and take the cared person as the identification category to obtain the walking state characteristics of the person; The specific process of step S3 is as follows: S31, classifying and identifying the perspective-transformed RGB image of the person under care outdoors according to the set classification angle, i.e., estimating the viewing angle; S32. Calculate the importance weights of the person's appearance features and gait energy features: Among them, ω a is the importance weight of the appearance feature, ω b is the importance weight of the gait energy feature, and x is the estimated result of the perspective of the person outside the care center; S33. Establish a fusion model with importance weights, and then use the fusion model to perform target detection training for outdoor personnel under care. The expression of the fusion model is as follows: where ω c is the importance weight of the walking state feature. When x>135°, ω c =1, otherwise ω c =0,f cd (·) represents the encoder in the autoencoder, f c (M) is the walking state feature.

2. The method for re-identifying a person outside a cared area based on multiple fusion features as claimed in claim 1, characterized in that: The specific process of step S4 is as follows: S41, using a bottom-up approach based on semantic segmentation to perform target classification detection of outdoor caregivers, outdoor staff and outdoor personnel being cared for the current outdoor image; S42, then perform image search on the images classified as outdoor personnel under care using a K-nearest neighbor algorithm, and compare the search results with the recognition structure obtained by the fusion model.

Images