Data processing method and device, electronic equipment and computer storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and computer storage medium. The data processing method comprises: obtaining a first data model and a sample image, wherein the first data model comprises a machine learning model; obtaining multi-modal supervision data corresponding to the sample image according to at least the spatio-temporal information of the sample image; and training the first data model according to at least the sample image and the multi-modal supervision data to obtain a second data model. The data processing method can improve the training effect of the data model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to a data processing method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] With the continuous development of artificial intelligence and machine learning technology, machine learning technology is increasingly widely used in daily life. A neural network model trained using machine learning technology can quickly and efficiently complete some daily tasks. For example, in the field of pedestrian re-identification of video monitoring, a trained pedestrian re-identification model can be used to process monitoring videos, thereby realizing functions such as tracking the moving track of a specific person. However, the application scenarios of existing pedestrian re-identification models are relatively limited. For example, a pedestrian re-identification model trained for an A scene cannot be directly used in a B scene.

[0003] To solve the above problems, in the prior art, a sample image used for training in the B scene is manually labeled to obtain a batch of supervised data for training, and then the supervised data manually labeled is used to train a pedestrian re-identification model suitable for the B scene. That is, in the prior art, there is a common problem that machine learning models need manual labeling of supervised data, resulting in high training cost, great difficulty, and difficulty in popularization. SUMMARY

[0004] Therefore, embodiments of the present application provide a data processing scheme to at least partially solve the above problems.

[0005] According to a first aspect of embodiments of the present application, a data processing method is provided, comprising: obtaining a first data model and a sample image, wherein the first data model comprises a machine learning model; obtaining multi-modal supervised data corresponding to the sample image according to at least spatial and temporal information of the sample image; and training the first data model according to at least the sample image and the multi-modal supervised data, to obtain a second data model.

[0006] According to a second aspect of embodiments of the present application, a data processing method is provided, comprising: obtaining multi-modal supervised data corresponding to a sample image according to at least spatial and temporal information of the sample image; and training a data model to be trained according to at least the sample image and the multi-modal supervised data, to obtain a target data model, wherein the data model to be trained comprises a machine learning model.

[0007] According to a third aspect of the embodiments of the present application, a data processing method is provided, comprising: receiving a model training request for requesting training of a first data model, which is sent by a client through calling a preset training interface; obtaining a sample image and the first data model for training according to the model training request; obtaining multi-modal supervision data corresponding to the sample image through the data processing method of the first aspect, and training the first data model using the sample image and the multi-modal supervision data.

[0008] According to a fourth aspect of the embodiments of the present application, a data processing apparatus is provided, comprising: a first obtaining module configured to obtain a first data model and a sample image, wherein the first data model comprises a machine learning model; a second obtaining module configured to obtain multi-modal supervision data corresponding to the sample image according to at least spatial-temporal information of the sample image; and a training module configured to train the first data model according to at least the sample image and the multi-modal supervision data, to obtain a second data model.

[0009] According to a fifth aspect of the embodiments of the present application, a data processing apparatus is provided, comprising: a third obtaining module configured to obtain multi-modal supervision data corresponding to a sample image according to at least spatial-temporal information of the sample image; and a second training module configured to train a data model to be trained according to at least the sample image and the multi-modal supervision data, to obtain a target data model, wherein the data model to be trained comprises a machine learning model.

[0010] According to a sixth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, the executable instruction causes the processor to perform operations corresponding to the data processing method of the first aspect, the second aspect or the third aspect.

[0011] According to a seventh aspect of the embodiments of the present application, a computer storage medium is provided, which stores a computer program, the program is executed by a processor to implement the data processing method of the first aspect, the second aspect or the third aspect.

[0012] According to the data processing solution provided by an embodiment of the present invention, multimodal supervision data is mined from sample images based on the spatiotemporal information of the sample images, so that the supervision data used to train the first data model does not need to be manually labeled, thereby reducing the difficulty and labor cost in the process of converting the data model from the source scene to the target scene, and achieving the purpose of training the data model without manually labeling the sample images of the target scene. Moreover, since multimodal supervision data is used during training, the effect of training the data model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0014] Figure 1a This is a flowchart of a data processing method according to the first embodiment of the present invention;

[0015] Figure 1b for Figure 1a A schematic diagram of an example scenario in the illustrated embodiment;

[0016] Figure 1c for Figure 1a A schematic diagram of a usage scenario in the illustrated embodiment;

[0017] Figure 2 This is a flowchart of a data processing method according to the second embodiment of the present invention;

[0018] Figure 3 This is a flowchart of a data processing method according to Embodiment 3 of the present invention;

[0019] Figure 4a This is a flowchart of a data processing method according to a fourth embodiment of the present invention;

[0020] Figure 4b for Figure 4a A schematic diagram of an example scenario in the illustrated embodiment;

[0021] Figure 5 This is a flowchart of a data processing method according to a fifth embodiment of the present invention;

[0022] Figure 6a This is a flowchart of a data processing method according to a sixth embodiment of the present invention;

[0023] Figure 6bThis is a schematic diagram of a connection between a SaaS platform and a client according to a sixth embodiment of the present invention;

[0024] Figure 7 This is a structural block diagram of a data processing device according to Embodiment 7 of the present invention;

[0025] Figure 8 This is a structural block diagram of a data processing device according to an eighth embodiment of the present invention;

[0026] Figure 9 FIG. 1 is a schematic structural diagram of an electronic device according to a ninth embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0028] The specific implementation of the embodiment of the present invention is further described below with reference to the accompanying drawings of the embodiment of the present invention.

[0029] Example 1

[0030] 1 , there is shown a flowchart of the steps of a data processing method according to a first embodiment of the present invention.

[0031] In this embodiment, the data processing method can be applied to the training of a person re-identification model, enabling the training of a person re-identification model for a source scene without manually labeling sample images in the target scene, so that it can be used in the target scene. The data processing method can be configured and executed by the server (the server includes the server and the cloud).

[0032] Of course, in other embodiments, the data processing method can be applied to other appropriate fields, such as training other neural network models for image processing. Moreover, the data processing method can be executed by any appropriate execution entity, such as a terminal device, and this embodiment does not limit this.

[0033] Step S102: Acquire a first data model and a sample image.

[0034] During the first data model conversion process, the source scene may refer to the scene containing the training samples used when the first data model was initially trained. For example, if the first data model A is a person re-identification model trained using surveillance images of city A, then the source scene is city A.

[0035] In contrast to the source scene, the target scene refers to the scene to which the first data model is applied. For example, if first data model A is used to perform pedestrian re-identification in surveillance video from city B, the target scene is city B. However, due to differences in architectural style, clothing style, and other factors from city A, simply applying first data model A to the target scene may not produce the desired effect.

[0036] The first data model can be any appropriate machine learning model (e.g., a neural network model). For example, the first data model can be the aforementioned person re-identification model, or a neural network model for face recognition. A person re-identification model refers to a neural network model that can perform surveillance video retrieval for specific pedestrians.

[0037] For first data models with different functions, different types of images can be used as sample images. For example, if the first data model is a person re-identification model, the sample image can be a surveillance image of a surveillance video, which includes a captured human body image of a pedestrian.

[0038] Step S104: obtaining multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image.

[0039] Spatiotemporal information includes at least the acquisition time and location of the sample image. For example, when a surveillance camera acquires surveillance images, it records the timestamp (i.e., acquisition time) of each surveillance image, as well as the latitude and longitude information of each surveillance camera. This latitude and longitude information represents the acquisition location of all surveillance images captured by that surveillance camera.

[0040] In multimodal supervision data, multimodality refers to the combination of information from multiple modalities, such as the combination of facial information and spatiotemporal information. Multimodal supervision data includes first supervision data and / or second supervision data. The first supervision data can be strong supervision supervision data, which means that the accuracy of the pseudo labels established based on multimodal information of these supervision data is very high (such as 100% or meets the conditions of approximately 100%). The second supervision data can be weak supervision supervision data, which means that the accuracy of the pseudo labels established based on multimodal information is relatively low (usually less than 100% or does not meet the conditions of approximately 100%).

[0041] Those skilled in the art can obtain multimodal supervision data in any appropriate manner as needed.

[0042] Taking the acquisition of strongly supervised multimodal supervised data as an example, the acquisition process is as follows:

[0043] For a target sample image Q in the sample images, a sample image W containing the same face as the target sample image Q is determined according to the face information in each sample image. Since the sample image W and the target sample image Q contain the same face, it can be considered that the sample image W and the target sample image Q contain the same person, and therefore the pseudo label of the sample image W is a positive sample.

[0044] According to the spatio-temporal information of the sample images, a sample image E is determined, which is the same in time as the target sample image Q and the distance between the collection positions is greater than a set distance threshold (the set distance threshold can be determined as needed). Since a person cannot be in two places at the same time, it can be considered that the sample image E and the target sample image Q cannot contain the same person, and therefore the pseudo label of the sample image E is a negative sample. In this way, the multi-modal supervised data with strong supervision can be obtained according to the sample image W and the sample image E.

[0045] Of course, in other embodiments, other ways can be used to obtain multi-modal supervised data, which are not limited in this embodiment.

[0046] Step S106: training the first data model according to at least the sample images and the multi-modal supervised data to obtain a second data model.

[0047] After obtaining the multi-modal supervised data, the sample images are used to train the first data model. During the training process, the multi-modal supervised data corresponding to the sample images is used as supervision, so as to calculate the loss value, and the parameters in the first data model are updated according to the loss value, so as to obtain a second data model applicable to the target scene.

[0048] The implementation process of the data processing method will be described below in combination with a specific use scenario:

[0049] In this use scenario, the first data model is a pedestrian re-identification model, and the sample images in the target scene are monitoring images in the monitoring video of city b.

[0050] In this use scenario, the monitoring images in the monitoring video obtained from city b are used as sample images. The first data model can be a pedestrian re-identification model trained based on the monitoring images of city a.

[0051] As shown in Figure 1b and Figure 1c In this use scenario, the data processing method is configured in the server, and the server obtains the first data model and the sample images.

[0052] For a sample image, at least according to its spatio-temporal information, the multi-modal supervised data of the sample image is obtained.

[0053] As shown in

[0054] The total number of sample images is N. For the i-th sample image I i , according to the sample image I i The face information of the sample image I is determined i The remaining N-1 sample images are the same as the sample image I i Sample images containing the same face (denoted as I j ), since the human body images in the sample images containing the same face are most likely the same person, the sample image I j That is a positive sample, the data model for the sample image I i and I j The loss function of the output feature information should be small.

[0055] In addition to obtaining positive samples, we can also i The acquisition time and acquisition position of the sample images are determined, and the acquisition time and acquisition position are the same, but the distance between the acquisition positions is greater than the set distance threshold. Since it is impossible for a person to appear at two locations far apart (such as 1 km) at the same time, the two sample images far apart will not include the same person, so the two form a negative sample pair and are negative samples of each other.

[0056] Multimodal supervision data is formed by positive samples and negative samples. The first data model can be trained using sample images and multimodal supervision data to obtain a second data model. The second data model can be applied to pedestrian re-identification in surveillance images of city b.

[0057] The trained second data model allows different test images to be input into the data model. If two test images contain the same person, the feature information corresponding to the two output test images will be spatially closer. This allows for person re-identification by inputting an image of a specific subject into the data model to obtain its feature information. This feature information can then be used to retrieve the feature information of other surveillance images, thereby obtaining surveillance images containing the specific subject and identifying the specific subject's movement trajectory.

[0058] Through this embodiment, multimodal supervision data is mined from the sample images based on the spatiotemporal information of the sample images, so that the supervision data used to train the first data model does not need to be manually labeled, thereby reducing the difficulty and labor cost in the process of converting the data model from the source scene to the target scene, and achieving the purpose of training the data model without manual labeling of the sample images. Moreover, since multimodal supervision data is used during training, the effect of training the data model can be improved.

[0059] Example 2

[0060] Reference Figure 2 , shows a step flow chart of a data processing method according to embodiment 2 of the present invention.

[0061] In this embodiment, the data processing method includes the aforementioned steps S102 to S106. In this embodiment, an implementation method for obtaining first supervisory data in multimodal supervisory data is exemplified, and step S104 includes sub-steps S1041 and S1042.

[0062] Sub-step S1041: performing face recognition on the sample image to obtain face information corresponding to the sample image.

[0063] For example, any existing appropriate face recognition technology is used to perform face recognition on all sample images to obtain face information of each sample image.

[0064] Sub-step S1042: determining first supervisory data corresponding to the sample image based on the facial information and the spatiotemporal information of the sample image.

[0065] In this embodiment, the first supervisory data is strongly supervised multimodal supervisory data.

[0066] In a specific implementation, sub-step S1042 includes:

[0067] Sub-step I: Based on the facial information of the sample images, determine a positive sample set consisting of sample images containing the same face.

[0068] For example, based on the recognized face information, the sample images belonging to the same person are determined. Any two of these sample images can form a positive sample pair, which is recorded as

[0069] For example, the sample image I i And sample image I j Both include face A, then the two form a positive sample pair, recorded as in represents the kth positive sample pair, I i Represents the i-th sample image, I j represents the jth sample image.

[0070] Sample Image I n And sample image I m Both include face B, then the two form a positive sample pair, recorded as in represents the kth positive sample pair, I n Represents the nth sample image, I mRepresents the mth sample image.

[0071] These positive sample pairs constitute the positive sample set.

[0072] Sub-step II: Determine a negative sample set consisting of sample images that meet set conditions based on the spatiotemporal information of the sample images.

[0073] In a specific implementation, sub-step II can be implemented as follows: determining a target sample image from the sample images; from the sample images other than the target sample image, determining as negative sample images the sample images whose acquisition time is the same as that of the target sample image and whose acquisition positions are at a distance greater than a set distance threshold; and determining a negative sample set corresponding to the target sample image based on the negative sample images.

[0074] The target sample image can be any sample image among the sample images. That is, sub-step II can be performed for each sample image to obtain a negative sample pair related thereto, and further obtain a negative sample set.

[0075] In this embodiment, each sample image has corresponding spatiotemporal information, including the acquisition time and location. The acquisition time can be determined based on the timestamp of the sample image. The acquisition location can be determined based on the surveillance camera that captured the sample image. For example, each surveillance camera has a corresponding serial number and latitude and longitude information.

[0076] For the target sample image, the latitude and longitude information of the target surveillance camera can be determined according to the number of the target surveillance camera that collected it, and then the surveillance cameras whose distance from the target surveillance camera is greater than the set distance threshold (such as one kilometer) can be found. The distance between the sample images collected by these surveillance cameras (for the convenience of description, they are recorded as sample images in set1) and the target sample image is greater than the set distance threshold.

[0077] Then, from the sample images in set1, determine the sample image whose acquisition time is the same as the acquisition time of the target sample image and record it as The people included in these sample images are different from those in the target sample images, so they are negative samples of the target sample images. The target sample image and any negative sample can form a negative sample pair denoted as It can be expressed as in, represents the kth negative sample pair, I i represents the target sample image, represents the nth negative sample.

[0078] Since the speed of human movement is limited, as long as an appropriate distance threshold is set for different human movement modes (such as cycling, walking, running, etc.), it is ensured that the negative samples found are different from the people included in the target sample image.

[0079] According to the negative sample pairs of each target sample image, a negative sample set corresponding to the target sample image can be determined.

[0080] Sub-step III: determining first supervisory data corresponding to the sample image based on the positive sample set and the negative sample set.

[0081] In this embodiment, in order to ensure that after the data model is trained using the first supervisory data, when the data model is used, after the human body image of the same person is input into the data model, the feature information outputted is relatively close in space, thereby realizing the tracking of a specific person, for a certain sample image, the sample image, a positive sample in the positive sample set of the sample image, and a negative sample in the negative sample set of the sample image are used to form the corresponding first supervisory data.

[0082] For example, for the sample image I i , and its corresponding first supervision data can be expressed as Among them, I j is a positive sample, These are negative samples. Because the reliability of these positive and negative sample pairs is very high, with an accuracy close to 100%, and in the process of selecting positive and negative sample pairs, in addition to using the human image information in the sample image, the facial information and spatiotemporal information are also used. Therefore, the first supervised data can be called strongly supervised multimodal supervised data.

[0083] During training, the triplet loss function can be used to train the data model using the first supervised data.

[0084] Through this embodiment, multimodal supervision data is mined from the sample images based on the spatiotemporal information of the sample images, so that the supervision data used to train the first data model does not need to be manually labeled, thereby reducing the difficulty and labor cost in the process of converting the data model from the source scene to the target scene, and achieving the purpose of training the data model without manual labeling of the sample images. Moreover, since multimodal supervision data is used during training, the effect of training the data model can be improved.

[0085] In addition, the use of facial information and spatiotemporal information ensures the accuracy of positive and negative samples, ensuring that the first supervised data is strong supervised data with an accuracy rate close to 100%, thereby improving the training effect of the data model.

[0086] Embodiment Three

[0087] Referring to Figure 3 FIG. 3 shows a flowchart of steps of a data processing method according to Embodiment Three of the present application.

[0088] In the present embodiment, the data processing method comprises the aforementioned steps S102-S106. In the present embodiment, step S104 can adopt the aforementioned implementation manners. In the present embodiment, an implementation manner of obtaining second supervision data in multi-modal supervision data is exemplified, i.e., the second supervision data is obtained through sub-steps S1043-S1045 comprised in step S104.

[0089] It should be noted that in the present embodiment, step S104 can only comprise sub-steps S1043-S1045. Alternatively, it can also comprise sub-steps S1041-S1045. In this case, sub-steps S1043-S1045 can be executed before, after or in parallel with sub-steps S1041 and S1042.

[0090] Sub-step S1043: performing feature extraction on the sample image using the first data model to obtain feature information for representing human image information.

[0091] The first data model can be the aforementioned data model pre-trained in the source scene. Alternatively, it can also be a data model that completes one or more training cycles in the target scene. For example, for the first data model A, in the first training cycle, after N sample images are used to train it, it is considered that the first data model A completes the first training cycle. In order to facilitate description, the data model that completes the first training cycle is denoted as A'. In the second training cycle, the training object becomes the first data model A'. That is, in different training cycles, the first data model used for feature extraction on the sample image can be different, which can be a data model trained in the previous training cycle.

[0092] Sub-step S1044: sorting the to-be-sorted feature information according to at least the spatio-temporal information of the sample image and the similarity between the feature information and the to-be-sorted feature to obtain a target sorting result.

[0093] In a specific implementation, sub-step S1044 comprises the following sub-steps:

[0094] Sub-step A: determining a target sample image from the sample images.

[0095] The target sample image can be any sample image. For each sample image, its corresponding target sorting result can be obtained through sub-step B.

[0096] Sub-step B: Sort the feature information to be sorted according to the similarity between the feature information of the target sample image and the feature to be sorted to obtain an initial sorting result.

[0097] The features to be sorted may be feature information of the remaining sample images except the target sample image. For example, if there are N sample images, of which the i-th sample image is the target sample image, then the remaining N-1 sample images are the remaining sample images.

[0098] For the target sample image, similarity can be determined by calculating the distance between the target sample image and each of the remaining sample images. The features to be sorted can then be sorted based on the similarity to obtain an initial sorting result. For example, the initial sorting result is sorted from high to low similarity. The higher the similarity, the higher the probability that the remaining sample images corresponding to the feature to be sorted and the target sample image contain the same person.

[0099] Of course, in other embodiments, the features to be sorted may be sorted in other ways, which is not limited in this embodiment.

[0100] Sub-step C: adjusting the initial sorting result at least according to the spatiotemporal information of the target sample image to obtain the target sorting result.

[0101] In a specific implementation, sub-step C includes the following process:

[0102] Process C1: performing a first adjustment on the initial sorting result according to the similarity between the facial information of the target sample image and the facial information of the sample image corresponding to the feature to be sorted, to obtain a first pre-adjusted sorting result.

[0103] Facial information can be obtained by performing facial recognition on the target sample image and the remaining sample images. The higher the similarity between the facial information of the target sample image and the remaining sample images, the higher the probability that the two images contain the same person. Therefore, performing a first adjustment (such as recalibration and sorting) on ​​the initial sorting results based on the facial information can make the sorting of the obtained first pre-adjusted sorting results more accurate, and the remaining sample images containing the same person as the target sample image can be sorted higher.

[0104] Process C2: performing a second adjustment on the first pre-adjusted sorting result according to the spatiotemporal information of the target sample image and the spatiotemporal information of the sample image corresponding to the feature to be sorted, to obtain a second pre-adjusted sorting result.

[0105] In addition to adjusting the sorting based on facial information, a second adjustment can be made to the sorting based on spatiotemporal information (i.e., recalibrating and re-sorting the second pre-adjusted sorting result), thereby utilizing spatiotemporal information to compensate for deficiencies in facial information and thereby improving sorting accuracy. When adjusting based on spatiotemporal information, the order of the features to be sorted for the remaining sample images that were acquired at the same time as the target sample image but at a greater distance from the target sample image can be shifted backward, as the likelihood of these remaining sample images containing the same person as the target sample image is low.

[0106] Process C3: performing a third adjustment on the second pre-adjusted sorting result according to the human attribute information of the target sample image and the human attribute information of the sample image corresponding to the feature to be sorted, to obtain the target sorting result.

[0107] To further improve sorting accuracy, in addition to adjusting the sorting based on facial and spatiotemporal information, adjustments can also be made based on body attribute information. Body attribute information includes appearance information, such as whether a hat is worn, hat color, clothing color, height, and so on. Using body attribute information can better adjust the sorting of features where faces are ambiguous or where spatiotemporal information cannot accurately determine whether they contain the same person, thereby increasing the accuracy of the target sorting results.

[0108] For example, for the target sample image, the faces in some of the remaining sample images are relatively blurred, and it is impossible to determine whether the target sample image and the remaining sample images contain the same person based on the facial information, and it is also impossible to accurately determine whether the two contain the same person based on the spatiotemporal information of the target sample image and the spatiotemporal information of the remaining sample images. The second pre-adjusted sorting result can be adjusted for the third time based on the human body attribute information (that is, the second pre-adjusted sorting result can be recalibrated and sorted) to obtain the target sorting result, so that the target sorting result is more accurate.

[0109] Sub-step S1045: Determine the target sorting result as the second supervision data.

[0110] The target ranking result after ranking adjustment is used as the second supervision data, that is, the ranking label of the target sample image. Since the accuracy of the target ranking result is not 100%, it is called weakly supervised multimodal training data.

[0111] The ranking loss function can then be used to train the data model using this weakly supervised multimodal data as supervision. This allows the trained data model to produce more accurate ranking results for test images. Specifically, images with a higher probability of containing the same person as the test image are ranked higher, enabling more accurate and rapid determination of the movement trajectory of a specific person.

[0112] Through this embodiment, multimodal supervision data is mined from the sample images based on the spatiotemporal information of the sample images, so that the supervision data used to train the first data model does not need to be manually labeled, thereby reducing the difficulty and labor cost in the process of converting the data model from the source scene to the target scene, and achieving the purpose of training the data model without manually labeling the sample images of the target scene. Moreover, since multimodal supervision data is used during training, the effect of training the data model can be improved.

[0113] In addition, the initial sorting results are adjusted using facial information and spatiotemporal information to obtain a more accurate target sorting result, which is used as weakly supervised multimodal supervision data, so that the sorting output by the second data model trained based on it is more accurate when used.

[0114] Example 4

[0115] Reference Figure 4a , shows a step flow chart of a data processing method according to embodiment 4 of the present invention.

[0116] In this embodiment, the data processing method includes the aforementioned steps S102 to S106. Step S104 can be implemented in any of the aforementioned ways. In this embodiment, before step S106, the method further includes steps S104a and S104b.

[0117] Step S104a: Using the first data model to perform feature extraction on the sample image.

[0118] It should be noted that this step can be performed at any appropriate time before step S106 , and can be performed before, after, or in parallel with step S104 .

[0119] The first data model may be the aforementioned data model pre-trained in the source scene. Alternatively, it may be a data model that has completed one or more training cycles in the target scene. For example, for the first data model A, in the first training cycle, after it is trained using N sample images, it is considered that the first data model A has completed the first training cycle. For the sake of convenience, the data model that has completed the first training cycle is recorded as A'. In the second training cycle, the training object becomes the first data model A'. That is to say, in different training cycles, the data model for feature extraction of sample images may be different, and it may be the first data model trained in the previous training cycle.

[0120] Step S104b: performing clustering processing on the feature information of the sample image to obtain the category to which the sample image belongs, and determining the category as the unimodal supervision data of the sample image.

[0121] Those skilled in the art may use any appropriate clustering method to perform clustering, such as K-means clustering algorithm, etc. The category obtained by clustering is the pseudo label corresponding to the sample image.

[0122] For example, if we cluster the feature information of N sample images and obtain K categories, and sample images 1, 2, and 5 belong to the same category, then the category they belong to is (denoted as label1). The pseudo label of sample image 1 is label1, which is the corresponding unimodal supervision data.

[0123] Since the reliability of feature information and clustering is relatively low, the accuracy cannot be guaranteed to be 100%, so it is called weakly supervised supervised data.

[0124] This unimodal supervised data can then be used to train the data model using the classification loss function.

[0125] After obtaining the supervision data, the first data model can be trained using the supervision data so that the trained second data model can adapt to the target scenario. In this embodiment, step S106 includes the following sub-steps:

[0126] Sub-step S1061: performing training weight configuration on the first supervisory data, the second supervisory data, and the unimodal supervisory data.

[0127] In order to improve the training effect, multiple supervisory data can act on the first data model at the same time, thereby making the trained first data model more effective. In this embodiment, the process of training using each supervisory data can be regarded as a sub-training framework, and the three sub-training frameworks can be carried out simultaneously. The training of the data model is a multi-task learning, and the three sub-training frameworks are used to update the same first data model at the same time.

[0128] To ensure that different supervisory data have different effects on the training results during this training process, training weights are configured for the first supervisory data, the second supervisory data, and the unimodal supervisory data. For example, the training weight for the first supervisory data is 0.7, the training weight for the second supervisory data is 0.2, and the training weight for the unimodal supervisory data is 0.1.

[0129] Of course, those skilled in the art can configure any appropriate training weights as needed, and this embodiment does not limit this.

[0130] Sub-step S1062: Use the sample image, the first supervised data with weight configuration, the second supervised data, the unimodal supervised data and the loss function corresponding to the supervised data to perform multi-task training on the first data model.

[0131] For example, a training process of a training cycle is: select one supervised data from the first supervised data, the second supervised data and the unimodal supervised data according to the training weight; if the selected supervised data is the first supervised data, the sample image corresponding to the selected first supervised data is used as input and input into the first data model; the first supervised data is used as supervision, the triplet loss loss function is used to calculate the loss value, and the parameters of the first data model are updated according to the loss value.

[0132] After completing a training cycle, the process of selecting a supervised data set from the first, second, and unimodal supervised data based on the training weights can be repeated until a training cycle is complete. The loss function used in each training cycle corresponds to the selected supervised data set. If the selected supervised data set is the first supervised data set, the loss function is the triplet loss function; if the selected supervised data set is the second supervised data set, the loss function is the ranking loss function; if the selected supervised data set is the unimodal supervised data set, the loss function is the classification loss function.

[0133] The following describes the implementation process of data processing based on a specific usage scenario:

[0134] In this usage scenario, the data model can be trained in the source scenario, and the first data model obtained through training is used as the initial model. Of course, other existing models can also be used, and this usage scenario does not limit this.

[0135] For the target scene to be transferred, obtain the corresponding sample image, such as the surveillance image in the surveillance video.

[0136] like Figure 4b As shown, in this usage scenario, in order to avoid the high cost, difficulty, and difficulty in scalability of manually labeling a batch of sample images for training in new scenes in the existing technology, the data processing method of this usage scenario mines more information from the sample images of the target scene as supervision data in the initial model trained on the source scene, achieving the purpose of automatically obtaining supervision data. This supervision data is then used to train and update the initial model, thereby obtaining a data model with better performance in the target scene.

[0137] In this usage scenario, three types of information are mined, namely, first supervisory data with strong supervision, second supervisory data with weak supervision, and unimodal supervisory data. Using these three types of supervisory data to train the first data model overcomes the problem in the prior art of using human body images in sample images to generate certain pseudo supervisory data to assist in training without labeling, resulting in the accuracy of the trained second data model being too low and difficult to meet actual usage requirements. It can also overcome the problem in the prior art of using human body images in the source scene to automatically generate fake human body images of the target scene for training of the target scene, which results in poor training results using fake sample images of the target scene.

[0138] Since the supervised data obtained by using multimodal information has higher accuracy, it helps to improve the effect of the trained data model, which is better than the training effect of using single-modal pseudo-labels.

[0139] The following combination Figure 4b Explain the process of obtaining supervised data and using supervised data for training:

[0140] like Figure 4b As shown, in this usage scenario, the first data model is a pedestrian re-identification model.

[0141] After obtaining a sample image of the target scene, the first aspect mines strongly supervised first supervision data based on the sample image, which integrates facial information and spatiotemporal information. Therefore, the obtained first supervision data is strongly supervised multimodal supervision data.

[0142] Specifically, for the sample images, face recognition technology is used to perform face recognition on all sample images of the target scene (i.e., pedestrian images contained in the surveillance video) to obtain face information. Using the face information, sample images belonging to the same person are found among these sample images and associated. These sample images belonging to the same person are used to form positive sample pairs. For example, sample image I i And sample image I j If the two sample images contain the same person, then they constitute a positive sample pair.

[0143] Since each sample image has the acquisition time and the corresponding number of the surveillance camera that captured it, and each surveillance camera has longitude and latitude information. Therefore, for each sample image, such as sample image I i The corresponding surveillance camera number can be determined, and then the latitude and longitude information can be obtained. The latitude and longitude information can be used to find all surveillance cameras that are more than one kilometer away from the surveillance camera, and these surveillance cameras are recorded as set1. From all the sample images collected in set1, find the one that matches the sample image I i The sample images with the same acquisition time are impossible to be the same as the sample image Ii Contains sample images of the same person. Use these sample images to create negative sample pairs, which can be recorded as

[0144] The positive sample pairs from face information and the negative sample pairs from spatiotemporal information are used to form strong multimodal supervised data. The training data can be formed by combining the sample images and the supervised data, which is expressed as The first data model is trained using the training data using the triplet loss function. Because the reliability of these sample pairs can be very high, with an accuracy close to 100%, it is called strongly supervised multimodal supervised data.

[0145] In the second aspect, second supervisory data, namely, weakly supervised multimodal supervisory data, is mined based on the sample image.

[0146] Specifically, based on a first data model trained using data of the source scene, feature extraction is performed on the sample image to obtain feature information.

[0147] The following process is performed on each sample image in the target scene to obtain the corresponding second supervision data.

[0148] For example, a target sample image is determined from the target scene, and the remaining sample images are the remaining sample images. The feature information of the target sample image and the features to be sorted of the remaining sample images are used to sort and obtain an initial sorting result.

[0149] The initial sorting result is recalibrated and sorted using the facial information of the target sample image to obtain a new and more accurate first pre-adjusted sorting result.

[0150] The first pre-adjusted sorting result is recalibrated and sorted using the spatiotemporal information of the target sample image to obtain a new and more accurate second pre-adjusted sorting result.

[0151] The second pre-adjusted sorting result is recalibrated and sorted using the human body attribute information of the target sample image to obtain a new and more accurate target sorting result.

[0152] The target ranking results are used as ranking labels for the target sample images. Subsequently, weakly supervised training data can be created based on the target sample images and the target ranking results. Because the accuracy of the target ranking results is not 100%, this is called weakly supervised training data. Training is performed using the ranking loss function and the resulting training data.

[0153] In a third aspect, weakly supervised single-modal supervised data is mined in the target scene. For example, a first data model trained using source scene data is used to extract features from sample images in the target scene, obtaining feature information. A clustering method is used to cluster the feature information, and the class to which the clustered feature information belongs is used as supervised data (i.e., pseudo labels).

[0154] The pseudo labels obtained by clustering and the sample images can be used to establish weakly supervised training data. Because the reliability of the feature information and the class to which the feature information belongs is relatively low, the accuracy cannot be guaranteed to be 100%, and therefore the training data is referred to as weakly supervised training data. The weakly supervised training data is used to train the data model using a classification loss function.

[0155] After the supervised data is obtained, each type of supervised data is regarded as a sub-training framework. The three sub-training frameworks are trained simultaneously, and the entire training process is regarded as a multi-task learning process, i.e., the three sub-training frameworks are trained simultaneously, and the same model is updated. The training weights are used to adjust the proportion of the influence of different training frameworks on the data model. Finally, a new second data model (e.g., a pedestrian re-identification model) for the target scene is obtained.

[0156] After a training cycle is completed, the trained first data model is used to replace the first data model of the source scene, and the above process is repeated until the required number of training cycles is completed to obtain the second data model.

[0157] In the above process, not only sample images (containing human body images) in the target scene are used to train the first data model, but also various modal information such as face information and spatiotemporal information are introduced to extract supervised data and form training data.

[0158] Furthermore, the supervised data obtained by various methods is modularized according to the accuracy and is divided into three different sub-training frameworks, and is fused together using a multi-task learning method to update the first data model. Compared with a model trained using only one type of supervised data, the effect is better, and manual labeling in the target scene is not required, and the training effect of the data model can be automatically improved.

[0159] In this embodiment, the multi-modal supervised data is mined from sample images according to spatiotemporal information and the like, so that the supervised data used to train the first data model does not need to be manually labeled, thereby reducing the difficulty and labor cost in the process of transferring the data model from the source scene to the target scene, achieving the purpose of training the data model without manual labeling of the sample images in the target scene, and because the multi-modal supervised data is used during training, the training effect of the data model can be improved.

[0160] Embodiment Five

[0161] Referring to Figure 5 FIG. 6 shows a flowchart of steps of a data processing method according to an embodiment of the present application.

[0162] In this embodiment, the data processing method comprises the following steps:

[0163] Step S502: Obtain multi-modal supervision data corresponding to the sample image according to the spatio-temporal information of the sample image.

[0164] Different sample images can be used for different use requirements. For example, for use requirements in a pedestrian re-identification scene, monitoring images can be used as sample images. For use requirements in an object recognition scene, images containing different objects can be used as sample images, and so on.

[0165] Taking the pedestrian re-identification scene as an example, the time information of the sample image at least includes the collection time and the collection location of the sample image. The collection time can be a timestamp created when the sample image is taken. The collection location can be the location of the image collection device that takes the sample image.

[0166] The multi-modal in the multi-modal supervision data means that multiple modal information is combined, such as the combination of face information and spatio-temporal information. The multi-modal supervision data includes first supervision data and / or second supervision data. The first supervision data can be strongly supervised supervision data, and strong supervision means that the pseudo-label accuracy of the supervision data based on multi-modal information is very high (such as 100% or meets the condition of approximately 100%). The second supervision data can be weakly supervised supervision data, and weak supervision means that the pseudo-label accuracy based on multi-modal information is relatively low (usually less than 100% or does not meet the condition of approximately 100%).

[0167] Those skilled in the art can obtain the multi-modal supervision data corresponding to the sample image in any appropriate manner, which is not limited in this embodiment.

[0168] Taking the obtaining of the first supervision data as an example, the first supervision data is used to indicate whether the same target object is contained in two sample images. For this purpose, according to the face information of the target sample image Q and the face information of the remaining sample images, the sample images containing the same face as the target sample image Q can be determined. Since the same face usually represents the same person, the sample images containing the same face and the target sample image Q form a positive sample pair, and the first supervision data of each sample image containing the same face is a positive sample.

[0169] Similarly, for the target sample image Q, based on the spatiotemporal information of the target sample image Q and the spatiotemporal information of the remaining sample images, it is possible to determine the sample images that were collected at the same time as the target sample image Q, but the distance between the collection positions is greater than the set distance threshold (the set distance threshold can be determined as needed). Since under normal circumstances, the same person cannot appear in two positions at the same time, the sample images whose collection positions have a distance greater than the set distance threshold form a negative sample pair with the target sample image Q, and the first supervisory data of the sample images whose collection positions have a distance greater than the set distance threshold is a negative sample.

[0170] In this way, since at least the spatiotemporal information of the sample image is integrated when determining the supervision data, multimodal supervision data can be obtained, which not only realizes the automatic acquisition of supervision data, but also does not require manual labeling, reducing the cost of obtaining supervision data.

[0171] Step S504: training the data model to be trained based on at least the sample image and the multimodal supervision data, and obtaining a target data model.

[0172] The data model to be trained includes a machine learning model. It can be a machine learning model of different forms for different use cases. For example, in a person re-identification scenario, it can be a convolutional neural network model, or a convolutional neural network model combined with an attention mechanism, etc.

[0173] In the process of training the data model to be trained using sample images, the multimodal supervision data corresponding to the sample images is used as supervision, and the loss value is calculated. Then, the parameters in the data model to be trained are updated according to the loss value to obtain the target data model that meets the requirements of the usage scenario.

[0174] For example, during training, sample images Q and W form a positive sample pair, and the corresponding first supervisory data indicates that sample image W is a positive sample. Sample images Q and W are then input into the data model to be trained, and the output data of the data model to be trained is obtained. A loss value is calculated based on the first supervisory data and the output data, and the parameters of the data model to be trained are adjusted based on the loss value. Other positive sample pairs are then input into the adjusted data model to be trained, and this process is repeated until the training termination condition is met and the target data model is obtained.

[0175] Optionally, in this embodiment, the target data model trained for the pedestrian re-identification scenario can be applied to the path recognition of the target object. For example, the following steps are performed to realize the recognition of the moving path of the target object.

[0176] Step S506: Acquire multiple frames of images in the video data, wherein at least some of the frames of images in the multiple frames contain the target object.

[0177] At least part of the multiple frames of images may be one or more frames. For example, if the video data is a surveillance video, one or more frames of images may capture the target object in the surveillance video.

[0178] Step S508: Use the target data model to perform object recognition on the multiple frames of images.

[0179] When performing object recognition, multiple frames of images are input into a target data model, and the target data model is used to identify a target frame image containing a target object as a recognition result.

[0180] Step S510: Determine the moving path of the target object according to the recognition result.

[0181] For each target frame image included in the recognition result, the acquisition positions of each target frame image are combined according to the acquisition time of each target frame image, thereby forming a moving path of the target object.

[0182] For example, the target frame images include image 1, image 2, image 3 and image 7, and the corresponding acquisition times are t1, t2, t3 and t7, respectively, and the corresponding acquisition positions are P1, P2, P3 and P2. The moving path of the target object determined based on the target frame images can be expressed as: P1-P2-P3-P2.

[0183] Through this data processing method, multimodal supervision data of sample images can be mined based on the spatiotemporal information of the sample images, so that the supervision data used to train the data model to be trained does not need to be manually labeled, thereby reducing the difficulty and labor cost of training the target data model, and achieving the purpose of training the data model without manual labeling of the sample images. Moreover, since multimodal supervision data is used during training, the effect of training the data model can be improved.

[0184] Example 6

[0185] Reference Figure 6a , shows a step flow chart of a data processing method according to embodiment 6 of the present invention.

[0186] In this embodiment, the data processing method provided by the embodiment of the present invention is described by taking the deployment of the data processing method on the server side (such as the cloud or server or SaaS platform) and training the first data model according to the client request as an example.

[0187] The data processing method of this embodiment includes the following steps:

[0188] Step S602: Receive a model training request for requesting training of a first data model, which is sent by the client by calling a preset training interface.

[0189] Taking the method deployed on a SaaS platform as an example, the training interface can be an interface preset by the SaaS platform for receiving model training requests from the client. The interface can be configured in any appropriate form as needed, and this embodiment does not limit this.

[0190] The first data model may be the model described in any one of embodiments 1 to 4. The model training request may be a request in any appropriate form.

[0191] Step S604: According to the model training request, sample images for training are obtained.

[0192] The first data model can be deployed on the SaaS platform, or obtained by the SaaS platform from a client or a third party through the network, which is not limited in this embodiment.

[0193] For example, if the first data model is deployed on a SaaS platform, when obtaining sample images, the SaaS platform can receive a model training request from a client to train the first data model. Based on the model training request, the SaaS platform can retrieve the sample images for training locally from the SaaS platform. In this case, if the SaaS platform already has suitable sample images stored locally, they can be directly obtained, thereby improving the speed and efficiency of model training.

[0194] In another feasible approach, when the SaaS platform receives a model training request from a client to train the first data model, it can collect sample images for training from a third party based on the model training request. For example, the sample images can be obtained from a third-party website or a data interface provided by a third-party application. In this case, the SaaS platform obtains the sample images from a third party, eliminating the need for local storage and conserving storage resources on the SaaS platform.

[0195] In another feasible approach, when a SaaS platform receives a model training request from a client to train a first data model, the SaaS platform can obtain sample images for training from the client based on the model training request. In this case, the client stores sample images, and the SaaS platform obtains these sample images from the client to train a second data model that better meets the client's requirements.

[0196] Step S606: Obtain multimodal supervision data corresponding to the sample image, and use the sample image and the multimodal supervision data to train a first data model.

[0197] For example, the multimodal supervision data corresponding to the sample image can be obtained by any of the methods in Examples 1 to 4, and the first data model can be trained using the sample image and the multimodal supervision data to obtain the second data model, which will not be repeated here.

[0198] The following is an example of deploying the first data model on a SaaS platform to illustrate the above process. Figure 6b shown.

[0199] Figure 6b In the process, the client sends a model training request to the SaaS platform. After receiving the request, the SaaS platform obtains sample images for training the first data model from the local storage device through a processing device. Based on the obtained sample images, the SaaS platform obtains multimodal supervision data of the sample images based on at least the spatiotemporal information of the sample images. The first data model is trained using the sample images and multimodal supervision data. After completing the training of the first data model and obtaining the second data model, the SaaS platform sends a training completion message to the client. Subsequently, if necessary, the client can send video data or images to be detected to the SaaS platform to obtain the movement path of the corresponding target object.

[0200] In the above, the first data model is deployed on a SaaS platform as an example, but those skilled in the art should understand that the solution of this embodiment is also applicable to the case where the first data model is deployed on other forms of servers.

[0201] It can be seen that through this embodiment, the first data model and its training are deployed on the server side, and the server side trains the first data model according to the request of the client, thereby realizing the training of the first data model without any requirements for client resources or performance, and ensuring the training effect and efficiency.

[0202] Example 7

[0203] Reference Figure 7 , shows a structural block diagram of a data processing device according to embodiment 7 of the present invention.

[0204] In this embodiment, the data processing device includes: a first acquisition module 702, used to acquire a first data model and a sample image, wherein the first data model includes a machine learning model; a second acquisition module 704, used to obtain multimodal supervision data corresponding to the sample image based on at least the spatiotemporal information of the sample image; and a first training module 706, used to train the first data model based on at least the sample image and the multimodal supervision data to obtain a second data model.

[0205] Optionally, the second acquisition module 704 includes: a face recognition module 7041, used to perform face recognition on the sample image to obtain face information corresponding to the sample image; a determination module 7042, used to determine the first supervision data corresponding to the sample image based on the face information and the spatiotemporal information of the sample image.

[0206] Optionally, the determination module 7042 is used to determine a positive sample set consisting of sample images containing the same face based on the facial information of the sample image; determine a negative sample set consisting of sample images that meet set conditions based on the spatiotemporal information of the sample image; and determine the first supervisory data corresponding to the sample image based on the positive sample set and the negative sample set.

[0207] Optionally, the determination module 7042 is used to determine a target sample image from the sample images when determining a negative sample set consisting of sample images that meet set conditions based on the spatiotemporal information of the sample images; from sample images other than the target sample image, a sample image that has the same acquisition time as the target sample image and a distance between acquisition positions greater than a set distance threshold is determined as a negative sample image; and a negative sample set corresponding to the target sample image is determined based on the negative sample image.

[0208] Optionally, the determination module 7042 includes: a first feature extraction module 7042a, used to use the first data model to extract features from the sample image to obtain feature information used to characterize human body image information; a sorting module 7042b, used to sort the feature information to be sorted based on at least the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted, to obtain a target sorting result; and a supervision determination module 7042c, used to determine the target sorting result as the second supervision data.

[0209] Optionally, the sorting module 7042b is used to determine a target sample image from the sample images; sort the feature information to be sorted according to the similarity between the feature information of the target sample image and the feature to be sorted to obtain an initial sorting result; and adjust the initial sorting result at least according to the spatiotemporal information of the target sample image to obtain the target sorting result.

[0210] Optionally, the sorting module 7042b is configured to, when adjusting the initial sorting result according to the spatio-temporal information of the target sample image to obtain the target sorting result, performing first adjustment on the initial sorting result according to the similarity between the face information of the target sample image and the face information of the sample image corresponding to the feature to be sorted, to obtain a first pre-adjustment sorting result; performing second adjustment on the first pre-adjustment sorting result according to the spatio-temporal information of the target sample image and the spatio-temporal information of the sample image corresponding to the feature to be sorted, to obtain a second pre-adjustment sorting result; and performing third adjustment on the second pre-adjustment sorting result according to the body attribute information of the target sample image and the body attribute information of the sample image corresponding to the feature to be sorted, to obtain the target sorting result.

[0211] Optionally, the apparatus further includes a second feature extraction module 708 configured to extract features of the sample image using the first data model before the first training module 706 trains the first data model according to the sample image and the multi-modal supervision data to obtain a second data model; and a clustering module 710 configured to perform clustering processing on the feature information of the sample image to obtain a category to which the sample image belongs, and determine the category as the single-modal supervision data of the sample image.

[0212] Optionally, the multi-modal supervision data includes first supervision data and second supervision data; and the first training module 706 includes a weight configuration module 7061 configured to configure training weights for the first supervision data, the second supervision data and the single-modal supervision data; and a model first training module 7062 configured to perform multi-task training on the data model using the sample image, the first supervision data, the second supervision data, the single-modal supervision data and a loss function corresponding to the supervision data after the training weights are configured.

[0213] The data processing apparatus of this embodiment is used to implement the corresponding data processing method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here. In addition, the functions of each module in the data processing apparatus of this embodiment can be implemented by referring to the description of the corresponding part in the foregoing method embodiments, which will not be described here either.

[0214] Embodiment Eight

[0215] With reference to Figure 8 FIG. 8 shows a structural block diagram of a data processing apparatus according to Embodiment Eight of the present application.

[0216] In this embodiment, the data processing apparatus includes:

[0217] A third acquisition module 802 is configured to obtain multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image;

[0218] The second training module 804 is used to train the data model to be trained based on at least the sample image and the multimodal supervision data, and obtain a target data model, wherein the data model to be trained includes a machine learning model.

[0219] Optionally, the device further comprises:

[0220] A fourth acquisition module 806 is configured to acquire multiple frames of images from the video data, wherein at least some of the frames of images include the target object;

[0221] An object recognition module 808 is configured to perform object recognition on the multiple frames of images using the target data model;

[0222] The path determination module 810 is configured to determine the moving path of the target object according to the recognition result.

[0223] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the data processing device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0224] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the data processing device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0225] Embodiment 9

[0226] Reference Figure 9 , shows a schematic structural diagram of an electronic device according to embodiment 6 of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0227] like Figure 9 As shown, the electronic device may include: a processor (processor) 902 , a communication interface (Communications Interface) 904 , a memory (memory) 906 , and a communication bus 908 .

[0228] in:

[0229] The processor 902 , the communication interface 904 , and the memory 906 communicate with each other via a communication bus 908 .

[0230] The communication interface 904 is used to communicate with other electronic devices or servers.

[0231] The processor 902 is configured to execute the program 910 , and specifically to execute the relevant steps in the above-mentioned embodiment of the verification code generation method.

[0232] Specifically, the program 910 may include program codes, which include computer operation instructions.

[0233] Processor 92 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a smart device may be of the same type, such as one or more CPUs, or may be of different types, such as one or more CPUs and one or more ASICs.

[0234] The memory 906 is used to store the program 910. The memory 906 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0235] First embodiment:

[0236] Program 910 can specifically be used to enable the processor 902 to perform the following operations: obtain a first data model and a sample image, wherein the first data model includes a machine learning model; obtain multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image; and train the first data model based at least on the sample image and the multimodal supervision data to obtain a second data model.

[0237] In an optional embodiment, the program 910 is also used to enable the processor 902 to perform face recognition on the sample image to obtain face information corresponding to the sample image when obtaining multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image; and determine the first supervision data corresponding to the sample image based on the face information and the spatiotemporal information of the sample image.

[0238] In an optional embodiment, the program 910 is also used to enable the processor 902 to determine the first supervisory data corresponding to the sample image based on the facial information and the spatiotemporal information of the sample image, to determine a positive sample set consisting of sample images containing the same face based on the facial information of the sample image; to determine a negative sample set consisting of sample images that meet set conditions based on the spatiotemporal information of the sample image; and to determine the first supervisory data corresponding to the sample image based on the positive sample set and the negative sample set.

[0239] In an optional embodiment, the program 910 is also used to enable the processor 902 to determine a target sample image from the sample images when determining a negative sample set consisting of sample images that meet set conditions based on the spatiotemporal information of the sample images; from the sample images other than the target sample image, the sample images that have the same acquisition time as the target sample image and the distance between the acquisition positions is greater than the set distance threshold are determined as negative sample images; and a negative sample set corresponding to the target sample image is determined based on the negative sample images.

[0240] In an optional embodiment, the program 910 is also used to enable the processor 902 to use the first data model to extract features from the sample image when obtaining multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image, so as to obtain feature information used to characterize human body image information; sort the feature information to be sorted based at least on the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted, so as to obtain a target sorting result; and determine the target sorting result as the second supervision data.

[0241] In an optional embodiment, the program 910 is also used to enable the processor 902 to determine a target sample image from the sample image when sorting the feature information to be sorted based on at least the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted to obtain a target sorting result; sort the feature information to be sorted based on the similarity between the feature information of the target sample image and the feature to be sorted to obtain an initial sorting result; and adjust the initial sorting result based on at least the spatiotemporal information of the target sample image to obtain the target sorting result.

[0242] In an optional embodiment, the program 910 is also used to enable the processor 902 to adjust the initial sorting result at least based on the spatiotemporal information of the target sample image, and when obtaining the target sorting result, perform a first adjustment on the initial sorting result based on the similarity between the facial information of the target sample image and the facial information of the sample image corresponding to the feature to be sorted, to obtain a first pre-adjusted sorting result; perform a second adjustment on the first pre-adjusted sorting result based on the spatiotemporal information of the target sample image and the spatiotemporal information of the sample image corresponding to the feature to be sorted, to obtain a second pre-adjusted sorting result; perform a third adjustment on the second pre-adjusted sorting result based on the human attribute information of the target sample image and the human attribute information of the sample image corresponding to the feature to be sorted, to obtain the target sorting result.

[0243] In an optional embodiment, the program 910 is also used to enable the processor 902 to train the first data model based on at least the sample image and the multimodal supervision data to obtain the second data model, and then use the first data model to extract features from the sample image; perform clustering on the feature information of the sample image to obtain the category to which the sample image belongs, and determine the category as the unimodal supervision data of the sample image.

[0244] In an optional embodiment, the multimodal supervision data includes first supervision data and second supervision data; the program 910 is also used to enable the processor 902 to train the first data model based on at least the sample image and the multimodal supervision data to obtain the second data model, and to configure training weights for the first supervision data, the second supervision data and the unimodal supervision data; and use the sample image, the first supervision data with weight configuration, the second supervision data, the unimodal supervision data and the loss function corresponding to the supervision data to perform multi-task training on the data model.

[0245] Second embodiment:

[0246] Program 910 can specifically be used to enable the processor 902 to perform the following operations: obtain multimodal supervision data corresponding to the sample image at least based on the spatiotemporal information of the sample image; train the data model to be trained at least based on the sample image and the multimodal supervision data, and obtain a target data model, wherein the data model to be trained includes a machine learning model.

[0247] In an optional embodiment, the program 910 is also used to enable the processor 902 to obtain multiple frames of images in the video data, at least some of the frame images in the multiple frames contain the target object; use the target data model to perform object recognition on the multiple frames of images; and determine the movement path of the target object based on the recognition results.

[0248] Third embodiment:

[0249] Program 910 can specifically be used to enable the processor 902 to perform the following operations: receive a model training request sent by the client by calling a preset training interface to request training of a first data model; obtain a sample image for training according to the model training request; obtain multimodal supervision data corresponding to the sample image through a data processing method as in the first embodiment, and use the sample image and the multimodal supervision data to train the first data model.

[0250] In an optional embodiment, the program 910 is also used to enable the processor 902 to obtain the sample image locally from the SaaS platform according to the model training request when obtaining the sample image according to the model training request; or, according to the model training request, the SaaS platform obtains the sample image from a third party; or, according to the model training request, the SaaS platform obtains the sample image from the client.

[0251] The specific implementation of each step in program 910 can be found in the corresponding descriptions of the corresponding steps and units in the above-mentioned data processing method embodiment, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiment, and will not be repeated here.

[0252] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0253] The method according to the embodiment of the present invention described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the data processing method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the data processing method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the data processing method shown here.

[0254] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

[0255] The above implementation methods are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.

Claims

1. A data processing method, comprising: Acquire a first data model and a sample image, wherein the first data model includes a machine learning model; Obtaining multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image; Training the first data model based at least on the sample image and the multimodal supervision data to obtain a second data model; The method of obtaining multimodal supervisory data corresponding to the sample image based at least on the spatiotemporal information of the sample image includes: using the first data model to extract features from the sample image to obtain feature information used to characterize human body image information; sorting the feature information to be sorted based at least on the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted to obtain a target sorting result, where the feature to be sorted is feature information of the remaining sample images except the target sample image, and the target sample image is any sample image among the sample images; and determining the target sorting result as the second supervisory data.

2. The method according to claim 1, wherein The obtaining of multimodal supervision data corresponding to the sample image at least based on the spatiotemporal information of the sample image includes: Performing face recognition on the sample image to obtain face information corresponding to the sample image; Determine first supervisory data corresponding to the sample image based on the facial information and the spatiotemporal information of the sample image.

3. The method according to claim 2, wherein: The determining, based on the face information and the spatiotemporal information of the sample image, first supervisory data corresponding to the sample image includes: Determining a positive sample set consisting of sample images containing the same face according to the facial information of the sample images; Determining a negative sample set consisting of sample images that meet set conditions based on the spatiotemporal information of the sample images; Determine first supervisory data corresponding to the sample image according to the positive sample set and the negative sample set.

4. The method according to claim 3, wherein: The step of determining a negative sample set consisting of sample images that meet set conditions based on the spatiotemporal information of the sample images includes: determining a target sample image from the sample images; Determining, from the sample images other than the target sample image, a sample image that is acquired at the same time as the target sample image and whose acquisition position is at a distance greater than a set distance threshold as a negative sample image; A negative sample set corresponding to the target sample image is determined according to the negative sample image.

5. The method according to claim 1, wherein The step of sorting the feature information to be sorted based on at least the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted to obtain a target sorting result includes: determining a target sample image from the sample images; According to the similarity between the feature information of the target sample image and the feature to be sorted, the feature information to be sorted is sorted to obtain an initial sorting result; The initial sorting result is adjusted at least according to the spatiotemporal information of the target sample image to obtain the target sorting result.

6. The method according to claim 5, wherein: The adjusting the initial sorting result at least according to the spatiotemporal information of the target sample image to obtain the target sorting result includes: performing a first adjustment on the initial sorting result according to the similarity between the facial information of the target sample image and the facial information of the sample image corresponding to the feature to be sorted, to obtain a first pre-adjusted sorting result; Performing a second adjustment on the first pre-adjusted sorting result according to the spatiotemporal information of the target sample image and the spatiotemporal information of the sample image corresponding to the feature to be sorted, to obtain a second pre-adjusted sorting result; According to the human attribute information of the target sample image and the human attribute information of the sample image corresponding to the feature to be sorted, a third adjustment is performed on the second pre-adjusted sorting result to obtain the target sorting result.

7. The method according to claim 1, wherein Before training the first data model based on at least the sample image and the multimodal supervisory data to obtain the second data model, the method further includes: Performing feature extraction on the sample image using the first data model; Clustering is performed on the feature information of the sample image to obtain the category to which the sample image belongs, and the category is determined as the unimodal supervision data of the sample image.

8. The method according to claim 7, wherein: The multimodal supervisory data includes first supervisory data and second supervisory data; The step of training the first data model at least based on the sample image and the multimodal supervision data to obtain a second data model includes: Performing training weight configuration on the first supervisory data, the second supervisory data, and the unimodal supervisory data; Multi-task training is performed on the data model using the sample image, the first supervised data with weight configuration, the second supervised data, the unimodal supervised data, and the loss function corresponding to the supervised data.

9. A data processing method, comprising: Obtaining multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image; Training a data model to be trained based at least on the sample image and the multimodal supervision data, and obtaining a target data model, wherein the data model to be trained includes a machine learning model; The obtaining of multimodal supervisory data corresponding to the sample image based at least on the spatiotemporal information of the sample image comprises: extracting features from the sample image using the data model to be trained to obtain feature information for characterizing human body image information; sorting the feature information to be sorted based at least on the spatiotemporal information of the sample image and the similarity between the feature information and features to be sorted to obtain a target sorting result, wherein the features to be sorted are feature information of remaining sample images excluding a target sample image, and the target sample image is any sample image among the sample images; and determining the target sorting result as the second supervisory data; Acquire multiple frames of images from video data, wherein at least some of the frames of images in the multiple frames contain a target object; Performing object recognition on the multiple frames of images using the target data model; The moving path of the target object is determined according to the recognition result.

10. A data processing method, comprising: Receiving a model training request for requesting training of a first data model, sent by a client by calling a preset training interface; Acquiring sample images for training according to the model training request; The multimodal supervision data corresponding to the sample image is obtained by the data processing method according to any one of claims 1 to 8, and the first data model is trained using the sample image and the multimodal supervision data.

11. The method according to claim 10, wherein: The step of obtaining a sample image according to the model training request includes: Obtain sample images locally from the SaaS platform according to the model training request; or, According to the model training request, the SaaS platform obtains sample images from a third party; or, According to the model training request, the SaaS platform obtains sample images from the client.

12. A data processing device comprising: A first acquisition module is configured to acquire a first data model and a sample image, wherein the first data model includes a machine learning model; a second acquisition module, configured to obtain multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image; A first training module, configured to train the first data model based at least on the sample image and the multimodal supervision data to obtain a second data model; The method of obtaining multimodal supervisory data corresponding to the sample image at least based on the spatiotemporal information of the sample image includes: using the first data model to extract features of the sample image to obtain feature information used to characterize human body image information; sorting the feature information to be sorted based on at least the spatiotemporal information of the sample image and the similarity between the feature information and the features to be sorted to obtain a target sorting result, where the features to be sorted are feature information of the remaining sample images except the target sample image, and the target sample image is any sample image among the sample images; and determining the target sorting result as the second supervisory data.

13. A data processing device comprising: a third acquisition module, configured to obtain multimodal supervision data corresponding to the sample image based at least on the spatiotemporal information of the sample image; a second training module, configured to train a data model to be trained based at least on the sample image and the multimodal supervisory data, and obtain a target data model, wherein the data model to be trained includes a machine learning model; obtaining multimodal supervisory data corresponding to the sample image based at least on the spatiotemporal information of the sample image, including: extracting features from the sample image using the data model to be trained to obtain feature information for characterizing human body image information; sorting the feature information to be sorted based at least on the spatiotemporal information of the sample image and the similarity between the feature information and features to be sorted to obtain a target sorting result, wherein the features to be sorted are feature information of the remaining sample images except the target sample image, and the target sample image is any sample image among the sample images; and determining the target sorting result as the second supervisory data; a fourth acquisition module, configured to acquire a plurality of frames of images from the video data, wherein at least some of the frames of images contain the target object; An object recognition module, configured to perform object recognition on the multiple frames of images using the target data model; The path determination module is used to determine the moving path of the target object according to the recognition result.

14. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the data processing method according to any one of claims 1 to 8, or the data processing method according to claim 9, or the operation corresponding to the data processing method according to claim 10 or 11.

15. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the data processing method according to any one of claims 1 to 8, or the data processing method according to claim 9, or the data processing method according to claim 10 or 11.

Images