Data processing method, training method, device, equipment and medium

Abstract

The present disclosure provides a data processing method, device, equipment, storage medium and program product, which relate to technical fields such as artificial intelligence or finance. The data processing method includes: obtaining M images collected by N cameras, wherein the M images include an image of a target user; inputting each of the M images into a pedestrian re-identification model according to the time sequence of collection; using the pedestrian re-identification model to confirm the identity and behavior status of the target user based on a first sequence feature, wherein the pedestrian re-identification model performs feature fusion on the features of each image based on the time series to obtain the first sequence feature. The present disclosure also provides a training method, device, equipment, storage medium and program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence or finance, and more specifically to a data processing method, training method, device, equipment, medium and program product. Background Art

[0002] Sometimes, people need to register their identities in work or daily life. For example, in scenarios such as counting conference participants, counting social or entertainment event participants, registering people entering and leaving a specific venue, user sign-in, or student attendance, there is a need to confirm and register the identities of relevant personnel. In related technologies, relevant personnel may be required to register manually, such as by handwriting or by punching in via mobile phone. Relevant personnel may also be required to perform face recognition for identity confirmation and registration.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the prior art:

[0004] Whether it is manual registration or face recognition, users tend to forget. In addition, the manual registration method is cumbersome, and the face recognition method can only confirm the identity of the user, but cannot confirm the user's behavior status. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a data processing method, training method, device, equipment, medium and program product that can automatically confirm the identity and behavior status of a user at the same time.

[0006] One aspect of an embodiment of the present disclosure provides a data processing method, including: acquiring M images captured by N cameras, wherein the M images include an image of a target user, and N and M are integers greater than or equal to 2; inputting each of the M images into a person re-identification model according to a time sequence of acquisition; and using the person re-identification model to confirm the identity and behavior status of the target user based on a first sequence feature, wherein the person re-identification model performs feature fusion on the features of each image based on the time series to obtain the first sequence feature.

[0007] According to an embodiment of the present disclosure, the N cameras are set at N positions, and obtaining M images captured by the N cameras includes: numbering the N cameras based on spatial information of the N positions, wherein each of the numbers corresponds to one of the N positions; and obtaining the M images from the N cameras in sequence according to the order of the numbers.

[0008] According to an embodiment of the present disclosure, the sequentially acquiring the M images from the N cameras includes: acquiring a first image captured by the camera corresponding to the 1st number, and a first timestamp of the first image; setting an initial value of i to 2, and looping the following operations until an image captured by the camera corresponding to the Nth number is acquired, specifically including: acquiring a second image captured by the camera corresponding to the i-th number after a predetermined time period of the first timestamp, and a second timestamp of the second image; setting the second timestamp as the new first timestamp, and setting i=i+1.

[0009] According to an embodiment of the present disclosure, every two adjacent numbered cameras include a first camera and a second camera, and the method also includes determining the predetermined time period, specifically including: for every two adjacent numbered cameras, detecting the image of the target user and the movement status of the target user from the image captured by the first camera; and determining the predetermined time period based on spatial information between the first camera and the second camera, and the movement status.

[0010] According to an embodiment of the present disclosure, the behavior state includes a first check-in state and a second check-in state, and confirming the identity and behavior state of the target user includes: after confirming the identity of the target user, when confirming that the target user moves along a first trajectory based on the first sequence feature, confirming that the target user is in the first check-in state; or when confirming that the target user moves along a second trajectory based on the first sequence feature, confirming that the target user is in the second check-in state.

[0011] According to an embodiment of the present disclosure, the pedestrian re-identification model includes an LSTM network, wherein the LSTM network is a long short-term memory neural network, and the pedestrian re-identification model obtains the first sequence feature by performing feature fusion on the features of each image based on the time series, including: inputting the features of each image into the LSTM network; performing time-progressive feature fusion on the features of each image according to the time series using the LSTM network; and obtaining the first sequence feature based on the output of the LSTM network.

[0012] According to an embodiment of the present disclosure, the features of each image include a first image feature of the target user and a second image feature of the background of the target user in each image. Before the features of each image are fused, the steps include: fusing the first image feature and the second image feature of each image through the LSTM network.

[0013] According to an embodiment of the present disclosure, the LSTM network includes a first LSTM network and a second LSTM network, and the pedestrian re-identification model is trained and obtained in the following manner, including: acquiring M images of a first user and M images of a second user according to a collected time series, wherein the first user is the same as or different from the second user; using the first LSTM network, performing time-progressive feature fusion on the features of each image of the first user based on the time series to obtain a second sequence feature; using the second LSTM network, performing time-progressive feature fusion on the features of each image of the second user based on the time series to obtain a third sequence feature; and constructing a loss function based on the second sequence features and the third sequence features to train the pedestrian re-identification model.

[0014] According to an embodiment of the present disclosure, the pedestrian re-identification model also includes a CNN network, the CNN network is a convolutional neural network, the CNN network includes a first CNN network and a second CNN network, and the method also includes: inputting M images of the first user into the first CNN network to obtain features of each image of the first user; inputting M images of the second user into the second CNN network to obtain features of each image of the second user.

[0015] Another aspect of an embodiment of the present disclosure provides a training method for a pedestrian re-identification model, wherein the pedestrian re-identification model includes an LSTM network, which is a long short-term memory neural network. The LSTM network includes a first LSTM network and a second LSTM network. The method includes: acquiring M images of a first user and M images of a second user according to a collected time series, wherein the first user is the same as or different from the second user; using the first LSTM network, performing time-progressive feature fusion on the features of each image of the first user based on the time series to obtain a second sequence feature; using the second LSTM network, performing time-progressive feature fusion on the features of each image of the second user based on the time series to obtain a third sequence feature; and constructing a loss function based on the second sequence feature and the third sequence feature to train the pedestrian re-identification model.

[0016] Another aspect of the disclosed embodiment provides a data processing device, including: a first acquisition module, an image input module and a user confirmation module. The first acquisition module is used to acquire M images captured by N cameras, wherein the M images include an image of a target user, and N and M are integers greater than or equal to 2. The image input module is used to input each of the M images into a person re-identification model according to a time sequence of acquisition. The user confirmation module is used to use the person re-identification model to confirm the identity and behavior status of the target user based on a first sequence feature, wherein the person re-identification model fuses the features of each image based on the time series to obtain the first sequence feature.

[0017] Another aspect of the disclosed embodiment provides a training device for a pedestrian re-identification model, wherein the pedestrian re-identification model includes an LSTM network, which is a long short-term memory neural network, and the LSTM network includes a first LSTM network and a second LSTM network. The device includes a second acquisition module, a first fusion module, a second fusion module, and a model training module. The second acquisition module is used to acquire M images of a first user and M images of a second user according to a collected time series, and the first user is the same as or different from the second user. The first fusion module is used to use the first LSTM network to perform time-progressive feature fusion on the features of each image of the first user based on the time series to obtain a second sequence feature. The second fusion module is used to use the second LSTM network to perform time-progressive feature fusion on the features of each image of the second user based on the time series to obtain a third sequence feature. The model training module is used to construct a loss function based on the second sequence feature and the third sequence feature to train the pedestrian re-identification model.

[0018] Another aspect of an embodiment of the present disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0019] Another aspect of the embodiments of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method as described above.

[0020] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0021] One or more of the above-mentioned embodiments have the following advantages or beneficial effects: it can at least partially solve the problem that manual registration is cumbersome and face recognition cannot confirm the user's behavior status, by obtaining M images taken across cameras between N cameras, and inputting the M images into the pedestrian re-identification model in time series, so as to facilitate the fusion of the features of each image based on the time series to obtain the first sequence features, wherein the first sequence features include the feature information of the target user and the feature information of time and / or space. The fused first sequence features can simultaneously confirm the identity and behavior status of the target user to improve the efficiency and accuracy of user identity registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 The application scenario diagram of the data processing method according to the embodiment of the present disclosure is schematically shown;

[0024] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0025] Figure 3 A flowchart schematically shows a data processing method according to another embodiment of the present disclosure;

[0026] Figure 4 A flowchart of acquiring M images according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 5 A flowchart of sequentially acquiring M images from N cameras according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 6 Schematically shows a flow chart for determining a predetermined time period according to an embodiment of the present disclosure;

[0029] Figure 7 Schematically shows a flow chart of obtaining first sequence features according to an embodiment of the present disclosure;

[0030] Fig. 8A The structure diagram of the pedestrian re-identification model in the training process according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 8B The structure diagram of the LSTM sub-network according to the embodiment of the present disclosure is schematically shown;

[0032] Fig. 9A flowchart of training a pedestrian re-identification model according to an embodiment of the present disclosure is schematically shown;

[0033] Fig.10 A flowchart of training a target detection model according to an embodiment of the present disclosure is schematically shown;

[0034] Fig.11 The structure block diagram of the data processing device according to the embodiment of the present disclosure is schematically shown;

[0035] Fig.12 A schematic diagram of a training device according to an embodiment of the present disclosure is shown; and

[0036] Fig.13 A block diagram of an electronic device suitable for implementing a data processing method or a training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0038] In the scenario where identity registration is required, user sign-in is taken as an example. In the related technology, for example, users use traditional sign-in methods such as manual card swiping or mobile phone application sign-in registration to register. Another example is that users use face recognition to sign-in. In fact, when users enter and exit the office, such as when leaving get off work, they may easily forget to sign in, causing losses to users, and later investigation will cause extra waste of manpower and material resources.

[0039] In addition, face recognition is easily affected by light, angle or distance, and can only passively wait for the user to enter a specific area for identity confirmation, and cannot determine the user's behavior status. For example, after the user completes the work clocking in after face recognition, and then leaves the office, face recognition is not performed at this time, and face recognition is performed again at the end of the get off work day to complete the clocking out.

[0040] Embodiments of the present disclosure provide a data processing method, a training method, an apparatus, a device, a medium, and a program product. The data processing method includes: obtaining M images captured by N cameras, wherein the M images include an image of a target user, and N and M are integers greater than or equal to 2. Each of the M images is input into a person re-identification model according to the time sequence of the acquisition. The person re-identification model is used to confirm the identity and behavior status of the target user based on the first sequence feature, wherein the person re-identification model fuses the features of each image based on the time series to obtain the first sequence feature.

[0041] According to an embodiment of the present disclosure, M images taken across N cameras are obtained, and the M images are input into a pedestrian re-identification model in a time series, so that the features of each image are fused based on the time series to obtain a first sequence feature, wherein the first sequence feature includes feature information of the target user and feature information of time or space. The fused first sequence features can simultaneously confirm the identity and behavior status of the target user, so as to improve the efficiency and accuracy of user identity registration.

[0042] It should be noted that the data processing method, training method, device, equipment, medium and program product provided in the embodiments of the present disclosure can be used in artificial intelligence, machine learning in image recognition related aspects, and can also be used in various fields other than artificial intelligence and machine learning technology, such as the financial field, etc. The application fields of the data processing method, training method, device, equipment, medium and program product provided in the embodiments of the present disclosure are not limited.

[0043] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision or application of user personal information involved complies with the provisions of relevant laws and regulations, takes necessary confidentiality measures, and does not violate public order and good morals.

[0044] Figure 1 The application scenario diagram of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0045] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first outdoor camera 111, a second outdoor camera 112, a first indoor camera 113 and a second indoor camera 114 (hereinafter collectively referred to as cameras), a network 140 and a server 150. Among them, the first user 131, the second user 132, and the third user 133 (hereinafter collectively referred to as users) move toward the building 120, and the fourth user 134 moves away from the building 120. The network 140 is used to provide a medium for a communication link between the cameras 111, 112, 113, 114 and the server 150. The network 140 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. Among them, the first indoor camera 113 and the second indoor camera 114 can be respectively set at different locations in the building 120, such as different floors.

[0046] The cameras 111, 112, 113, 114 can respectively capture videos in corresponding areas. For example, when the users 131, 132, and 133 walk through the shooting area of ​​any camera, their images can be collected. Then, the cameras 111, 112, 113, 114 can interact with the server 150 through the network 140 to receive or send videos or images, etc. The cameras 111, 112, 113, 114 can be surveillance cameras, or various electronic devices with camera functions, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc.

[0047] The server 150 may be a server that provides various services, such as processing videos or images sent by cameras 111, 112, 113, and 114, and obtaining training samples to train a pedestrian re-identification model, or using the trained pedestrian re-identification model to confirm the user's identity and behavior status. In some embodiments, the user may use the above electronic device to interact with the server 150 to send instructions for training the model or to send instructions for identity registration. The background management server may analyze and process the received user request and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the electronic device.

[0048] It should be noted that the data processing method or training method provided in the embodiment of the present disclosure can generally be executed by the server 150. Accordingly, the data processing device or training device provided in the embodiment of the present disclosure can generally be set in the server 150. The data processing method or training method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 150 and can communicate with the above-mentioned camera and / or server 150. Accordingly, the data processing device or training device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the above-mentioned camera and / or server 150 and can communicate with the server 150.

[0049] It should be understood that Figure 1 The number of cameras, buildings, users, networks and servers in the embodiment is only for illustration. Any number of cameras, buildings, users, networks and servers may be provided as required.

[0050] The following will be based on Figure 1 The scene described by Figure 2 to Figure 10 The data processing method and training method of the disclosed embodiment are described in detail.

[0051] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0052] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S230.

[0053] In operation S210, M images captured by N cameras are obtained, wherein the M images include an image of a target user, and N and M are integers greater than or equal to 2.

[0054] Reference Figure 1 , the N cameras may include cameras 111 to 114, and the M images may include images collected from the process of users 131, 132, and 133 entering the shooting area of ​​camera 114 from camera 111. At least one image is obtained from each camera. Each image may include at least one user, among which there may be a target user.

[0055] In operation S220, each of the M images is input into a person re-identification model according to a time sequence of acquisition.

[0056] For example, the timestamp of each camera capturing an image can be obtained, and M images can be arranged according to the timestamp, and then input according to the time sequence of the capture. For user images captured across multiple cameras, the target user's image is obtained from multiple cameras, and the user's features are obtained, which provides more comprehensive local detail information of pedestrians from multiple angles. At the same time, for different degrees of occlusion, the user's identity information can be more effectively determined. The user's movement trajectory is confirmed in combination with the time series, which facilitates the confirmation of the user's behavior status.

[0057] Take user 133 as an example. He walks from the position of camera 111 to the position of camera 114. With the change of time and space, the time when cameras 111 to 114 collect images of user 133 is different. Therefore, M images can be sequentially input into the pedestrian re-identification model according to the time sequence of collection.

[0058] In operation S230, the identity and behavior status of the target user are confirmed based on the first sequence features using a person re-identification model, wherein the person re-identification model fuses the features of each image based on a time series to obtain the first sequence features.

[0059] According to an embodiment of the present disclosure, for example, an image containing user 133 is obtained from each of cameras 111 to 114, and the pedestrian re-identification model can fuse the features of each image in a time-progressive manner, so that it can recognize the hidden feature expression patterns of different pedestrians and recognize the identity and behavior status of the target user. Therefore, the first sequence features obtained contain user information and time information of multiple images, which can confirm the identity and behavior status of the target user.

[0060] In the related art, face recognition is easily affected by light and distance, so it is necessary to purchase and install face recognition equipment and set a special face recognition area to ensure accurate recognition results. The data processing method of the embodiment of the present disclosure can use existing monitoring equipment to obtain M images taken across N cameras, and input the M images into the pedestrian re-identification model in time series, so as to fuse the features of each image based on the time series to obtain the first sequence features, wherein the first sequence features include the information of the target user and the time information. The fused first sequence features can simultaneously confirm the identity and behavior status of the target user, so as to improve the efficiency and accuracy of user identity registration.

[0061] Figure 3 The flowchart schematically shows a data processing method according to another embodiment of the present disclosure.

[0062] like Figure 3As shown, the data processing method of this embodiment includes operations S210 to S220, and may also include operations S310 to S330. Among them, operations S210 to S220 can refer to the above Figure 1 The relevant description is not repeated here.

[0063] In operation S310 , a person re-identification model is used to confirm the identity of a target user based on a first sequence of features.

[0064] In some embodiments, M registered images of the target user can be acquired in advance, and the registered sequence features can be obtained. When the identity and behavior status of the user are identified, the similarity between the first sequence feature and the registered sequence feature is calculated to output the classification result. Specifically, for example, in the scene of user check-in, when user 133 enters the venue, he moves from camera 111 to the shooting area of ​​camera 114, and M registered images and corresponding registered sequence features are collected. Correspondingly, when user 133 leaves the venue, he moves from camera 114 to the shooting area of ​​camera 111, and M registered images and corresponding registered sequence features are collected. The registered sequence features are stored in the registration database in correspondence with the identification of user 133. So that when confirming the identity and behavior status, search from the registration database. If the identity and behavior status of user 133 are identified, the check-in record is automatically updated.

[0065] In operation S320 , when it is confirmed based on the first sequence feature that the target user moves along the first trajectory, it is confirmed that the target user is in the first sign-in state.

[0066] In operation S330 , when it is confirmed based on the first sequence feature that the target user moves along the second trajectory, the target user is confirmed to be in the second sign-in state.

[0067] According to the embodiments of the present disclosure, in order to prevent the cameras 111 and 112 from being unable to effectively obtain the user's image in bad weather, such as rainy and snowy weather, so that the pedestrian re-identification model cannot extract sufficient time information from the images captured by cameras 111 to 114. In the case where the technology for confirming the identity of the target user is relatively mature, the identity information of the target user can be strengthened and mined to obtain the first sequence features for identity recognition. Then, the time information is strengthened and mined based on M pictures to obtain new first sequence features. Reference Figure 1 The process of user 133 entering building 120 is moving along the first trajectory (the time when cameras 111 to 114 capture user 133 is from early to late), and the process of user 134 moving away from building 120 is moving along the second trajectory (the time when camera 114 captures user 134 is the earliest, and the time when camera 111 captures user 134 is the latest).

[0068] According to an embodiment of the present disclosure, for example, in a scenario of user attendance, for example, user 134 first enters the office to complete the clock-in, and then leaves the office in building 120. Two records can be formed to store the attendance status of user 134. If user 134 waits until the end of get off work time to complete the clock-in, two records will also be formed. After the end of the working day, the user's attendance status can be determined based on the earliest and latest two check-in records of the working day. The time interval between each two adjacent check-in records can also be counted. If the time interval exceeds the specified time (for example, the time interval between leaving and entering is too long), an alarm can be issued to the relevant personnel.

[0069] Regarding the user's attendance problem, the relevant technology can only automatically identify the user's identity, but cannot determine whether the user is on duty or off duty. Considering that different cameras have certain sequence information, the images of the user appearing in different cameras contain time connections, so the input images are the images of the user taken by different cameras on his attendance track.

[0070] According to the embodiments of the present disclosure, when the identity of the target user is effectively confirmed, the user's behavior status can also be confirmed at the same time, so as to identify and count the user's behavior. For example, when predicting a user, the user's identity and attendance status can be predicted based on multiple images with a time series, and the prediction results can be used to form an attendance sequence. If there is an entry during working hours and a departure during off-duty hours in the distribution area of ​​the attendance sequence, it means that the person's attendance is complete and his attendance information can be updated.

[0071] Therefore, the use of pedestrian re-identification technology can obtain data input within a certain time series from the location where the user enters the collection area through cameras deployed in multiple areas. By enhancing the spatiotemporal continuity of the data, the user's identity information can be identified, and different check-in states can be predicted based on the time series to maintain user information.

[0072] It should be noted that although the above describes the various operations of the method in a specific order, the embodiments of the present disclosure are not limited thereto, and the above operations can be performed in other orders as needed. For example, step S320 and step S330 are different situations for the same target user and can be performed separately.

[0073] Figure 4 The flowchart of acquiring M images in operation S210 according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 4As shown, the acquisition of M images in this embodiment includes operations S410 to S420. N cameras are set at N positions, each position corresponds to a different area, and the areas captured by each camera are different.

[0075] In operation S410, N cameras are numbered based on spatial information of the N positions, wherein each number corresponds to one of the N positions.

[0076] According to an embodiment of the present disclosure, the spatial information may include coordinate information, distance information, height information, etc. Figure 1 , cameras 111, 112, 113, 114 are respectively arranged at 4 positions. The cameras 111, 112, 113, 114 can be numbered as 1, 2, 3, and 4, respectively.

[0077] In operation S420, M images are acquired from N cameras in sequence according to the order of the numbers.

[0078] According to an embodiment of the present disclosure, for example, during the working hours, the image of user 133 is first obtained from the image captured by camera 111, and the time stamp is 8:05, and then the image of user 133 is obtained from cameras 112, 113, and 114 in sequence, for example, images with time stamps of 8:06, 8:08, and 8:15 are obtained. During the off-duty hours, four images with time stamps of 6:00, 6:06, 6:08, and 6:15 can be obtained from cameras 114, 113, 112, and 111, respectively.

[0079] According to the embodiments of the present disclosure, numbering each camera is helpful to quickly obtain the target user's travel path. Figure 1 Other paths can also be included by comparing the camera numbers on each path (such as Figure 1 1, 2, 3 and 4 belong to one path), which is beneficial for directly acquiring the image from the next numbered camera of the corresponding path after acquiring the target image from the first camera, without traversing all cameras, thereby improving the efficiency of acquiring the image of the target user.

[0080] Figure 5 The flowchart of sequentially acquiring M images from N cameras in operation S420 according to an embodiment of the present disclosure is schematically shown.

[0081] like Figure 5 As shown, the sequential acquisition of M images from N cameras in this embodiment includes operations S510 to S540.

[0082] In operation S510, a first image captured by a camera corresponding to number 1 and a first timestamp of the first image are obtained. The number 1 may be the number of the initial position camera in different time periods, for example, during working hours, the number 1 is camera number 111. During off-duty hours, the number 1 is camera number 4 114.

[0083] According to an embodiment of the present disclosure, N cameras can record videos from their respective shooting areas during the acquisition process. When acquiring the image captured by the camera No. 1, firstly, video data can be acquired respectively, and image frames at fixed intervals (such as 1 second) can be output to form a data set. Then, the target detection model is used to determine the image frame containing the target user as the first image.

[0084] After operation S510 is performed, the initial value of i is set to 2, and operations S520 to S540 are performed in a loop until an image captured by the camera corresponding to the Nth number is obtained.

[0085] In operation S520, a second image captured by the camera corresponding to the i-th number after a predetermined time period of the first timestamp and a second timestamp of the second image are acquired.

[0086] In operation S530, it is determined whether an image captured by the camera corresponding to the Nth number is acquired. If so, the current loop ends. If not, operation S540 is executed.

[0087] In operation S540 , the second timestamp is used as a new first timestamp, and i=i+1.

[0088] According to the embodiments of the present disclosure, the amount of video data recorded by each camera is relatively large, but in fact only a few image frames may meet the requirements. Figure 1 If the target detection model is required to detect all video data collected by each camera to obtain images containing the target user, the detection speed will be slow, wasting system performance. Therefore, using the data processing method of the embodiment of the present disclosure, it is possible to determine when the target user appears in the shooting area of ​​a certain camera according to the user's movement state, so that the target detection model can perform target detection in a targeted manner and reduce the amount of data processing.

[0089] Figure 6 The flowchart of determining a predetermined time period according to an embodiment of the present disclosure is schematically shown.

[0090] like Figure 6 As shown, determining the predetermined time period in this embodiment includes performing operations S610 to S620 on every two adjacent cameras. Wherein, every two adjacent cameras include a first camera and a second camera. The first camera may be a camera that acquires an image first.

[0091] In operation S610, an image of a target user and a moving state of the target user are detected from an image captured by a first camera.

[0092] In operation S620, a predetermined time period is determined based on space information between the first camera and the second camera, and a movement state.

[0093] According to an embodiment of the present disclosure, taking camera 111 and camera 112 as an example, the spatial information between the two can be distance information. If the moving state of user 133 is a slow walking state, combined with the walking speed of an average person and the above distance information, it may take 5 minutes (only as an example) for it to move from camera 111 to the shooting area of ​​camera 112. If the moving state of user 132 is a running state, it may take 3 minutes for it to move from camera 111 to the shooting area of ​​camera 112. If the moving state of user 131 is a cycling state, it may take 1 minute for it to move from camera 111 to the shooting area of ​​camera 112. Among them, the moving state of the user can use the target detection model to identify the user's posture. Then, an image containing user 133 is obtained from camera 111, and the corresponding timestamp is time t, then an image 5 minutes after time t can be obtained from camera 112.

[0094] Taking camera 113 and camera 114 as an example, the spatial information between the two may be height information, and the predetermined time period from camera 113 to camera 114 may be determined based on whether the user is taking an elevator or climbing stairs. The state of taking an elevator may be determined based on the user's background, such as the background of an elevator door being opened, by extracting background feature information.

[0095] According to the embodiments of the present disclosure, in real life, the locations where the cameras are installed are different, so the spatial information between every two adjacent cameras is also different. By using the embodiments of the present disclosure, the predetermined time period can be flexibly determined, so that images containing the target user can be detected more quickly, avoiding the situation where a large amount of data is processed due to the use of a fixed predetermined time period.

[0096] Figure 7 The flowchart of obtaining the first sequence feature in operation S230 according to an embodiment of the present disclosure is schematically shown.

[0097] like Figure 7 As shown, the first sequence feature acquisition in this embodiment includes operations S710 to S730. The person re-identification model includes an LSTM (Long Short Term) network, wherein the LSTM network is a long short-term memory neural network.

[0098] In operation S710, the features of each image are input into the LSTM network.

[0099] According to an embodiment of the present disclosure, before operation S710, it may be determined that each image belongs to the same user image, specifically as follows:

[0100] After the target detection model obtains data, for example, there are three images at time T1 (such as image A, image B, and image C), three images at time T2 (such as image A, image B', and image C'), and three images at time T3 (such as image A", image B" and image C", and the three images at each time may include three users. Among them, although T1, T2, and T3 are time series, the target detection model may not be able to determine the sequence images of the same target user at time T1, T2, and T3. Therefore, the similarity of each image can be judged. Taking image A at time T1 as an example, the similarity of image A and image A', B', and C' at time T2 are judged respectively, and then sorted, and A and A' with the largest similarity are taken. Then the similarity between A and A", B", and C" at time T3 is calculated, and A and A' with the largest similarity are taken. Similarly, the same operation is performed on B and C. Finally, the sequence images of the same target user at time T1, T2, and T3 can be obtained.

[0101] In the related art, feedforward neural networks are usually used, but feedforward neural networks do not consider the correlation between data, and the output of the network is only related to the input of the current network. However, when confirming the user's behavior state, the output of the network at a certain moment is not only related to the input at the current moment, but also related to a certain moment or several moments before. However, due to the lack of memory ability, the feedforward neural network cannot pass the information of the previous moment to the back, and cannot handle this correlation well. The LSTM (Long Short Memory Network) of the disclosed embodiment can better memorize the previous input data and process the timing information between different images.

[0102] In operation S720, the features of each image are fused in a time-progressive manner according to the time sequence by using the LSTM network.

[0103] According to the embodiments of the present disclosure, referring to Figure 1, obtain 4 images collected by 4 cameras respectively. First, extract the user information and time information of each image to obtain the features of each image. Take the features of the first image as the first output, fuse the features of the first image with the features of the second image as the second output, fuse the second output with the features of the third image as the third output, and fuse the third output with the features of the fourth image as the fourth output. Through the above-mentioned time-progressive feature fusion, the first output to the fourth output that can reflect the temporal feature information are obtained.

[0104] In operation S730, a first sequence feature is obtained based on an output of the LSTM network.

[0105] According to an embodiment of the present disclosure, the pedestrian re-identification model can extract the underlying basic information of the sequence images of the same target user through the CNN network and the resnet50 neural network to obtain the features of each image. Then, it is input into the LSTM network. After the LSTM network strengthens the images with different temporal features, the feature output of the next layer is performed by memorizing the main features and forgetting irrelevant features, thereby forming multiple feature sets with contextual temporal information of the target user image. Finally, the output of the LSTM network layer is used as the input of a pooling layer for time accumulation and strengthening to obtain the first sequence feature.

[0106] According to an embodiment of the present disclosure, the features of each image include a first image feature of a target user and a second image feature of the background of the target user in each image. Before the features of each image are fused, the steps include: fusing the first image feature and the second image feature of each image through an LSTM network.

[0107] According to an embodiment of the present disclosure, each image includes a target user and the environment background where the target user is located. The target user may be cropped out of the image first, and a first image feature may be obtained from the cropped rectangular user image. Then, a second image feature may be extracted from the remaining environment background.

[0108] According to the embodiments of the present disclosure, referring to Figure 1 , the background when camera 111 captures the user is different from the background captured by camera 114, because the two are located at different positions. Therefore, the features representing the user information and the background information are first fused to extract the spatial features of each image. When performing time-progressive feature fusion, the time features of each image are further fused, so that the movement trajectory of the target user can be more accurately reflected.

[0109] Fig. 8AThe structure diagram of a pedestrian re-identification model during training according to an embodiment of the present disclosure is schematically shown. Figure 8B The structure diagram of the LSTM sub-network according to an embodiment of the present disclosure is schematically shown. Fig. 9 The flowchart of training a pedestrian re-identification model according to an embodiment of the present disclosure is schematically shown.

[0110] like Fig. 9 As shown, the data processing method of this embodiment includes operations S910 to S960. In which, the LSTM network includes a first LSTM network and a second LSTM network.

[0111] In operation S910, M images of a first user and M images of a second user are acquired according to a time sequence of acquisition, where the first user is the same as or different from the second user.

[0112] In operation S920, M images of the first user are input into a first CNN network to obtain features of each image of the first user.

[0113] In operation S930, the M images of the second user are input into the second CNN network to obtain features of each image of the second user.

[0114] like Fig. 8A As shown, the structure of the twin network can be used to train the pedestrian re-identification model. During the training process, the pedestrian re-identification model is set to have a first LSTM network (i.e., LSTM-1) and a second LSTM network (i.e., LSTM-2) with the same structure, as well as a first CNN network (i.e., CNN-1) and a second CNN network (i.e., CNN-2) with the same structure. Among them, LSTM-1 or LSTM-2) each includes a sub-long short-term memory network L1, L2 and L3, and the structure of each sub-network can be the same or different. CNN-1 or CNN-2 each includes a sub-convolutional neural network C1, C2 and C3. X(t-1), X(t), and X(t+1) represent images at time t-1, time t, and time t+1, respectively, and the image at each time includes an image of the target user. In some embodiments, the image at each time also includes a background image of the target user. In operation S940, the first LSTM network is used to perform time-progressive feature fusion of the features of each image of the first user based on the time series to obtain a second sequence feature.

[0115] Let's take L2 in LSTM-1 as an example, combined with Figure 8B , describing in detail the process of performing time-progressive feature fusion on the features of each image of the first user based on the time series.

[0116] like Figure 8B As shown, L2 may include a forget gate ft , input gate i t , candidate cell information and output gate o t .in, represents vector multiplication, represents the vector sum, σ represents the Logistic function. Cell state C t-1 is the internal state of the LSTM sub-network (such as L1) at the previous moment, h t-1 is the external state of the LSTM subnetwork (such as L1) at the previous moment. Cell state C t It is the internal state of the LSTM sub-network (such as L2) at the current moment, and records the historical information up to the current moment. t is the external state of the LSTM sub-network (such as L2) at the current moment.

[0117] First, based on h t-1 and X(t) to calculate the forget gate f t , where the forget gate f t By viewing h t-1 The information of X(t) is used to output a vector of 0-1, where the values ​​of 0-1 represent C t-1 How much information to keep or discard in . 0 means discard, 1 means keep.

[0118] Then, the input gate i is calculated t and candidate cell states Input Gate i t By viewing h t-1 The information of X(t) is used to decide which information to update. Candidate cell states Using h t-1 and X(t) are obtained by the tanh function.

[0119] Then, through the forget gate f t With C t-1 Perform vector multiplication operation through input gate i t and candidate cell states Perform vector multiplication and vector addition on the above two results to obtain C t .

[0120] Next, based on h t-1 And X(t) calculate to get the output gate o t , where o t According to the input h t-1 To determine which features to output with X(t), the input can be passed through a sigmoid layer to obtain the judgment condition.

[0121] Finally, Ct After the tanh layer, a vector (such as a value between -1 and 1) is obtained, and the vector is multiplied by the output gate to obtain h t .

[0122] In operation S950, the features of each image of the second user are subjected to time-progressive feature fusion based on the time series by using the second LSTM network to obtain third sequence features.

[0123] In operation S960 , a loss function is constructed based on the second sequence features and the third sequence features to train a person re-identification model.

[0124] According to the embodiments of the present disclosure, referring to Fig. 8A , training a person re-identification model can do the following:

[0125] First, obtain sample data of multiple users, for example, obtain multiple groups of images for each user, each group of images contains multiple images collected in time series, and the selection includes data from multiple dates as much as possible to increase the effect of subsequent model training. In addition, for example, the training data set and the test data set are determined in a ratio of 70% and 30%. Specifically, in the process of making sample data of a certain user, obtain his photos from multiple camera videos (the photos include the user's full body data), set the same ID for the photos of the same person taken from different lenses, and construct the complete data set.

[0126] Then, data is annotated for each set of images using manual annotation, such as annotating the identity and behavior status of the user in each set of images.

[0127] Next, any two groups of images in the training data set are randomly paired to form multiple sample pairs, wherein each sample pair includes M images of a first user and M images of a second user, and the first user and the second user may be the same person or different persons.

[0128] Next, referring to operations S920 and S930, the two groups of images in the sample pair are input to CNN-1 and CNN-2 respectively.

[0129] According to an embodiment of the present disclosure, for example, at time t-1, time t, and time t+1, the images of the first user and the second user are input into CNN-1 and CNN-2, respectively. Among them, each sub-network C1, C2, and C3 extracts features from an input picture respectively. In some embodiments, CNN-1 and CNN-2 can also be trained first, so that after each sub-network extracts features, feature transfer is performed to extract the invariant features of the users in each image, and finally ensure that the output features belong to the same user. In some embodiments, each sub-network C1, C2, and C3 can be divided into three parts, a global feature extraction part, and two branch networks, which respectively extract features of the upper and lower parts of the image, and can extract features from multiple scales.

[0130] Next, the inputs of CNN-1 and CNN-2 are input to LSTM-1 and LSTM-2 respectively.

[0131] According to the embodiments of the present disclosure, for example, the image features output by C1, C2 and C3 (e.g., the feature data after the user features, time features and background features are combined) are respectively used as the inputs of L1, L2 and L3. Taking LSTM-1 as an example, L1 processes the image features at time t-1 to obtain C t-1 、h t-1 As the input of L2, and h t-1 As the input of the pooling layer (Temporal pooling-1). L2 fuses the image features of L1 and time t to obtain C t 、h t As the input of L3, and h t As the input of pooling layer-1. L3 fuses the image features of L2 and time t+1 t+1 , and used as the input of pooling layer-1 (Temporal pooling-1). The data processing process in LSTM-2 is the same as that of LSTM-1. As can be seen from the above, the output of the neural network corresponding to the previous moment is used as the input of the neural network at the next moment, and the output of the neural network at the next moment is obtained by integrating the image features of the current moment and the image features of all previous moments. Therefore, the multiple inputs of pooling layer-1 (Temporal pooling-1) have contextual information at the time series level.

[0132] Next, the outputs of LSTM-1 and LSTM-2 are temporally accumulated and strengthened in the pooling layer-1 (Temporal pooling-1) and the pooling layer-2 (Temporal pooling-2), and the pooling layer-1 (Temporal pooling-1) and the pooling layer-2 (Temporal pooling-2) output the second sequence feature (i.e., Seq.Feature-1) and the third sequence feature (i.e., Seq.Feature-2), respectively.

[0133] Next, a loss function is constructed based on Seq.Feature-1 and Seq.Feature-2 to train and determine the relevant parameters in the pedestrian re-identification model.

[0134] According to an embodiment of the present disclosure, for example, the classification result (user identity and behavior status) and the true label outputted by Seq.Feature-1 are obtained, and Identity loss-1 is constructed in combination with the cross entropy function. The classification result (user identity and behavior status) and the true label outputted by Seq.Feature-2 are obtained, and Identity loss-2 is constructed in combination with the cross entropy function. And Siameseloss is constructed based on the difference between Seq.Feature-1 and Seq.Feature-2. During the training process, the loss function is continuously iterated and optimized, for example, the sum of the loss functions Siamese Loss, Identity loss-1, and Identity loss-2 is minimized until the model converges, thereby completing the training of the pedestrian re-identification model.

[0135] Finally, the trained person re-identification model is tested using the test dataset.

[0136] According to an embodiment of the present disclosure, if the training is performed in a manner that the twin networks share parameter value changes, the neural networks on the left and right sides are the same. The trained person re-identification model may only include CNN-1, LSTM-1, and Seq.Feature-1, and be tested and used. In addition, images with a portrait judgment threshold of more than 95% (for example only) in the output results of the person re-identification model may be recorded. Since the number of times each user's image may appear in the model is inconsistent, the prediction results of each person's image are stored for subsequent queries.

[0137] According to the embodiments of the present disclosure, if training is performed in an asymmetric twin network, that is, optimization is performed based on the changes in Identityloss-1 and Siamese Loss, so that the Identity loss between the same person is minimized, the SiameseLoss is maximized, and the Siamese Loss between different people is maximized, and the parameters in CNN-1 and LSTM-1 are optimized and adjusted using back propagation. Based on the changes in Identityloss-2 and Siamese Loss, the parameters in CNN-2 and LSTM-2 are adjusted using back propagation. The trained person re-identification model includes the following: Fig. 8A The asymmetric twin network shown in Figure 1. During testing, the input sample pair includes two sets of images of the same user, which can be the same or different. The asymmetric twin network obtains Seq.Feature-1 and Seq.Feature-2 respectively, processes them and outputs the classification results.

[0138] It should be noted that, although the above describes the various operations of the method in a specific order, the embodiments of the present disclosure are not limited thereto, and the above operations can be performed in other orders as needed. For example, step S920 and step S930 can be performed at the same time, and step S940 and step S950 can also be performed at the same time. Fig. 8A The number of subnetworks, time points, and images in CNN-1, LSTM-1, CNN-2, and LSTM-2 is limited and can be selected according to actual needs.

[0139] In some embodiments of the present disclosure, the target detection model can also be trained so that the trained target detection model can output X(t-1), X(t), and X(t+1) that meet the requirements for use by the pedestrian re-identification model.

[0140] Fig.10 A flowchart of training a target detection model according to an embodiment of the present disclosure is schematically shown.

[0141] like Fig.10 As shown, the training target detection model of this embodiment may include operations S1010 to S1060. The target detection model may adopt the FastRCNN target detection network model.

[0142] In operation S1010, video data recorded by N cameras during multiple time periods by capturing images of multiple training user samples is obtained.

[0143] In operation S1020, the video data recorded by each camera is processed and output as a data set consisting of video frames at fixed intervals (eg, 1 second).

[0144] In operation S1030, it is determined whether it is in the training stage, and if so, operation S1040 is performed. If not, operation S1060 is performed. If it is not in the training stage, it means that it can be used normally.

[0145] In operation S1040, it is confirmed that it is in the training stage, and the labeled data with the target detection data frame is obtained. In some embodiments of the present disclosure, the images of the data set can be annotated in advance by manual annotation. The labelme software is used to annotate the image frames of the characters in the image, and the annotated data can be divided into two parts: the train training set and the test test set. The data of the training set is obtained in operation S1040.

[0146] In operation S1050, the FastRCNN target detection network model is trained. The FastRCNN target detection network is trained with the train training set, and after repeated iterative optimization, the model can obtain an effect of more than 97% (only as an example) on the test test set, ensuring that each video output image frame contains a pedestrian image in a single rectangular frame, and / or a cropped background image in each image.

[0147] In operation S1060, the trained FastRCNN target detection network model is obtained and put into use. For example, a video frame output from a video camera is obtained, and the FastRCNN target detection network model obtains a pedestrian image in a single rectangular frame, and / or a background image. Then, the outputted person rectangular image or background image is scaled to a standard input size to obtain X(t-1), X(t), and X(t+1), and input into the pedestrian re-identification network.

[0148] Based on the above data processing method and training method, the present disclosure also provides a data processing device and a training device.

[0149] Fig.11 The structure block diagram of the data processing device 1100 according to an embodiment of the present disclosure is schematically shown.

[0150] like Fig.11 As shown, the data processing device 1100 of this embodiment includes a first acquisition module 1110 , an image input module 1120 and a user confirmation module 1130 .

[0151] The first acquisition module 1110 may, for example, perform operation S210 to acquire M images captured by N cameras, wherein the M images include an image of a target user, and N and M are integers greater than or equal to 2.

[0152] According to an embodiment of the present disclosure, the first acquisition module 1110 may perform the aforementioned operations S410 to S420, S510 to S540, and S610 to S520, which are not described in detail herein.

[0153] The image input module 1120 may, for example, perform operation S220 to input each of the M images into the person re-identification model according to the time sequence of acquisition.

[0154] The user confirmation module 1130 may, for example, perform operation S230 to confirm the identity and behavior status of the target user based on the first sequence features using the person re-identification model, wherein the person re-identification model fuses the features of each image based on a time series to obtain the first sequence features.

[0155] According to an embodiment of the present disclosure, the user confirmation module 1130 may be used to execute the operations S310 to S330 and S710 to S730 described above, which will not be described in detail herein.

[0156] According to an embodiment of the present disclosure, the data processing device 1100 may further include a training module. The training module may perform the above operations S910 to S960.

[0157] Fig.12 The structural block diagram of the training device 1200 according to an embodiment of the present disclosure is schematically shown.

[0158] like Fig.12 As shown, the training device 1200 of this embodiment includes a second acquisition module 1210 , a first fusion module 1220 , a second fusion module 1230 and a model training module 1240 .

[0159] The second acquisition module 1210 may, for example, perform operation S910 to acquire M images of a first user and M images of a second user according to a time sequence of acquisition, where the first user is the same as or different from the second user.

[0160] The first fusion module 1220 may, for example, perform operation S940 to obtain second sequence features by using the first LSTM network to perform time-progressive feature fusion on the features of each image of the first user based on the time series.

[0161] The second fusion module 1230 may, for example, perform operation S950 to obtain third sequence features by using a second LSTM network to perform time-progressive feature fusion on the features of each image of the second user based on the time series.

[0162] The model training module 1240 may, for example, perform operation S960 to construct a loss function based on the second sequence features and the third sequence features to train a person re-identification model.

[0163] According to an embodiment of the present disclosure, any multiple modules in the data processing device 1100 or the training device 1200 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the data processing device 1100 or the training device 1200 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the data processing device 1100 or the training device 1200 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0164] Fig.13 A block diagram of an electronic device suitable for implementing a data processing method or a training method according to an embodiment of the present disclosure is schematically shown.

[0165] like Fig.13 As shown, the electronic device 1300 according to an embodiment of the present disclosure includes a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 into a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include an onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0166] In RAM 1303, various programs and data required for the operation of electronic device 1300 are stored. Processor 1301, ROM 1302 and RAM 1303 are connected to each other through bus 1304. Processor 1301 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1302 and / or RAM 1303. It should be noted that the program can also be stored in one or more memories other than ROM 1302 and RAM 1303. Processor 1301 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0167] According to an embodiment of the present disclosure, the electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to the bus 1304. The electronic device 1300 may further include one or more of the following components connected to the I / O interface 1305: an input portion 1306 including a keyboard, a mouse, etc.; an output portion 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1308 including a hard disk, etc.; and a communication portion 1309 including a network interface card such as a LAN card, a modem, etc. The communication portion 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed, so that a computer program read therefrom is installed into the storage portion 1308 as needed.

[0168] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0169] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1302 and / or RAM 1303 described above and / or one or more memories other than ROM 1302 and RAM 1303.

[0170] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method according to the embodiment of the present disclosure.

[0171] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1301. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0172] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1309, and / or installed from the removable medium 1311. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0173] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0174] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, Python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0175] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: Obtain M images captured by N cameras, wherein the M images include an image of a target user, and N and M are integers greater than or equal to 2; Input each of the M images into a person re-identification model according to a time sequence of acquisition; Using the person re-identification model to confirm the identity and behavior status of the target user based on the first sequence features, wherein the person re-identification model performs feature fusion on the features of each image based on the time series to obtain the first sequence features; The pedestrian re-identification model includes an LSTM network, wherein the LSTM network is a long short-term memory neural network, the LSTM network includes a first LSTM network and a second LSTM network, and the pedestrian re-identification model is obtained by training in the following manner, including: Acquire M images of a first user and M images of a second user according to a collected time sequence, where the first user is the same as or different from the second user; Using the first LSTM network, the features of each image of the first user are subjected to time-progressive feature fusion based on the time series to obtain second sequence features; Using the second LSTM network, the features of each image of the second user are subjected to time-progressive feature fusion based on the time series to obtain a third sequence feature; A loss function is constructed based on the second sequence features and the third sequence features to train the pedestrian re-identification model.

2. The method according to claim 1, wherein: The N cameras are set at N positions, and obtaining M images collected by the N cameras includes: Numbering the N cameras based on the spatial information of the N positions, wherein each of the numbers corresponds to one position among the N positions; The M images are acquired from the N cameras in sequence according to the sequence of the numbers.

3. The method according to claim 2, wherein: The sequentially acquiring the M images from the N cameras comprises: Obtain a first image captured by the camera corresponding to number 1, and a first timestamp of the first image; Let the initial value of i be 2, and loop through the following operations until the image captured by the camera corresponding to the Nth number is obtained, including: Acquire a second image captured by the camera corresponding to the i-th number after a predetermined time period of the first timestamp, and a second timestamp of the second image; Let the second timestamp be the new first timestamp, and let i=i+1.

4. The method according to claim 3, wherein: Every two cameras with adjacent numbers include a first camera and a second camera. The method further includes determining the predetermined time period, specifically including: for every two cameras with adjacent numbers, Detecting an image of the target user and a moving state of the target user from the image captured by the first camera; The predetermined time period is determined based on spatial information between the first camera and the second camera, and the movement state.

5. The method according to claim 1, wherein: The behavior status includes a first check-in status and a second check-in status, and the confirming the identity and behavior status of the target user includes: after confirming the identity of the target user, In a case where it is confirmed based on the first sequence feature that the target user moves along a first trajectory, confirming that the target user is in a first sign-in state; or When it is confirmed based on the first sequence feature that the target user moves along the second trajectory, the target user is confirmed to be in the second sign-in state.

6. The method according to claim 1, wherein: The pedestrian re-identification model performs feature fusion on the features of each image based on the time series to obtain the first sequence features, including: Input the features of each image into the LSTM network; Using the LSTM network, the features of each image are subjected to time-progressive feature fusion according to the time sequence; The first sequence feature is obtained based on the output of the LSTM network.

7. The method according to claim 6, wherein: The feature of each image includes a first image feature of the target user and a second image feature of the background of the target user in each image. Before the feature fusion of the features of each image, the method includes: The first image feature and the second image feature of each image are fused through the LSTM network.

8. The method according to claim 1, wherein: The pedestrian re-identification model further includes a CNN network, the CNN network is a convolutional neural network, the CNN network includes a first CNN network and a second CNN network, and the method further includes: Inputting the M images of the first user into the first CNN network to obtain features of each image of the first user; Input the M images of the second user into the second CNN network to obtain features of each image of the second user.

9. A method for training a person re-identification model, wherein: The pedestrian re-identification model includes an LSTM network, the LSTM network is a long short-term memory neural network, the LSTM network includes a first LSTM network and a second LSTM network, and the method includes: Acquire M images of a first user and M images of a second user according to a collected time sequence, where the first user is the same as or different from the second user; Using the first LSTM network, the features of each image of the first user are subjected to time-progressive feature fusion based on the time series to obtain second sequence features; Using the second LSTM network, the features of each image of the second user are subjected to time-progressive feature fusion based on the time series to obtain a third sequence feature; A loss function is constructed based on the second sequence features and the third sequence features to train the pedestrian re-identification model.

10. A data processing device, comprising: A first acquisition module is used to acquire M images captured by N cameras, wherein the M images include an image of a target user, and N and M are integers greater than or equal to 2; An image input module, used for inputting each of the M images into a pedestrian re-identification model according to a time sequence of acquisition; A user confirmation module, used to confirm the identity and behavior status of the target user based on the first sequence feature by using the pedestrian re-identification model, wherein the pedestrian re-identification model performs feature fusion on the features of each image based on the time series to obtain the first sequence feature; The pedestrian re-identification model includes an LSTM network, wherein the LSTM network is a long short-term memory neural network, the LSTM network includes a first LSTM network and a second LSTM network, and the pedestrian re-identification model is obtained by training in the following manner, including: Acquire M images of a first user and M images of a second user according to a collected time sequence, where the first user is the same as or different from the second user; Using the first LSTM network, the features of each image of the first user are subjected to time-progressive feature fusion based on the time series to obtain second sequence features; Using the second LSTM network, the features of each image of the second user are subjected to time-progressive feature fusion based on the time series to obtain a third sequence feature; A loss function is constructed based on the second sequence features and the third sequence features to train the pedestrian re-identification model.

11. A training device for a person re-identification model, wherein: The pedestrian re-identification model includes an LSTM network, which is a long short-term memory neural network. The LSTM network includes a first LSTM network and a second LSTM network. The device includes: A second acquisition module, configured to acquire M images of a first user and M images of a second user according to a time sequence of acquisition, wherein the first user is the same as or different from the second user; a first fusion module, configured to use the first LSTM network to perform time-progressive feature fusion on the features of each image of the first user based on the time series to obtain second sequence features; a second fusion module, configured to use the second LSTM network to perform time-progressive feature fusion on the features of each image of the second user based on the time series to obtain a third sequence feature; A model training module is used to construct a loss function based on the second sequence features and the third sequence features to train the pedestrian re-identification model.

12. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 9.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images