Model training method, crowd positioning method, device, equipment and medium

Abstract

The invention discloses a model training method and device, a crowd positioning method and device, equipment and a medium. The method comprises the steps of selecting a first view angle based on a training data set and adding the first view angle to a labeled view angle set; training an initial task model based on a labeling training data set for labeling the first labeled view angle set; selecting a second view angle based on the initial task model and the labeled view angle set, adding the second view angle to the labeled view angle set, and retraining the initial task model until a # imgabs0 # view angle is selected; and training a preset task model based on the labeled training data set labeled with the # imgabs1 # view angle. The prediction result of the initial task model is fully considered in the view angle selection process, so that the selected view angle can cover more crowds in the scene, the situation that people in the scene are missed is reduced, the detection accuracy of the target task model obtained through training is improved, the prediction accuracy of the multi-view task can be improved, and the user experience is improved. And the marking cost of the training task model is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a model training method, a crowd positioning method, a device, a device and a medium. Background Art

[0002] Multi-view crowd tasks (such as multi-view crowd counting and multi-view crowd localization) solve the large-scale scene coverage, severe occlusion and ambiguity problems encountered in single-view tasks by fusing multiple calibrated camera views at the same time. However, existing multi-view crowd tasks mainly focus on task models and generally use randomly sampled multi-view inputs. Randomly sampled multi-views may cause a large number of people in the scene to be omitted, resulting in inaccurate scene-level crowd prediction.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a model training method, a crowd positioning method, a device, equipment and a medium in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a model training method, wherein the model training method specifically includes: Acquire a training data set, wherein the training data set includes a plurality of multi-view images; Selecting a first perspective based on the training data set, and adding the first perspective to a labeled perspective set; Based on the labeled view set, the training data set is labeled with a crowd to obtain a labeled training data set, and an initial task model corresponding to a crowd task is trained based on the labeled training data set, wherein each labeled view in the labeled training data set carries a crowd label; Based on the initial task model and the labeled perspective set, a second perspective is selected, and the second perspective is added to the labeled perspective set, and the step of annotating the training data set based on the labeled perspective set to obtain an annotated training data set is re-executed until the first perspective is selected. Perspective, where is a positive integer; The first The perspective is added to a labeled perspective set, the training data set is crowd labeled based on the labeled perspective set to obtain a labeled training data set, and a preset task model corresponding to the crowd task is trained based on the labeled training data set to obtain a target task model corresponding to the crowd task.

[0006] The model training method, wherein the The perspective selection process specifically includes: Calculating a perspective evaluation value of each unlabeled perspective and a labeled perspective set on each frame of the multi-perspective image in the training data set, wherein the perspective evaluation value is determined based on scene coverage, average perspective distance, and perspective difference; Calculate the perspective evaluation value and the perspective evaluation value corresponding to each unlabeled perspective, and select the first one in the unlabeled perspective based on the perspective evaluation value and Perspective.

[0007] The model training method, wherein the initial task model corresponding to the task of training a group of people based on the labeled training data set specifically includes: For each multi-view image frame in the training data set, a pseudo-label perspective is selected for the multi-view image; The initial task model corresponding to the crowd task is trained based on the single-view images corresponding to each labeled view and pseudo-labeled view.

[0008] The model training method, wherein the initial task model corresponding to the single-view image training crowd task based on each labeled view and pseudo-label view specifically includes: Inputting the single-view images corresponding to each labeled view and pseudo-labeled view into an initial task model corresponding to the crowd task, and outputting a first predicted ground crowd density map through the initial task model; Obtaining a union of the visual fields of the labeled perspectives in the labeled perspective set to obtain a first joint visual field; Determine a first ground crowd density map based on the first predicted ground crowd density map and the first joint field of view; Constructing a first loss function term based on the first ground crowd density map and the crowd labels of the labeled viewpoints; The initial task model is trained based on the first loss function term.

[0009] The model training method, wherein the step of training a preset task model corresponding to a crowd task based on the labeled training data set to obtain a target task model corresponding to the crowd task specifically includes: For each multi-view image in the training dataset, select target perspectives, selected from the unlabeled perspectives other than the labeled perspective set pseudo-label perspectives, where , is a positive integer; Based on the Target perspective and The preset task model corresponding to the crowd task is trained with single-view images corresponding to the pseudo-labeled view.

[0010] The model training method, wherein the Target perspective and The preset task models corresponding to the single-view image training crowd task corresponding to the pseudo-label perspective include: Will Target perspective and The single-view image corresponding to the pseudo-label perspective is input into the preset task model, and the second predicted ground crowd density map is output through the preset task model; Obtain the union of the visual fields of each labeled perspective to obtain a first joint visual field, and Target perspective and The union of the fields of view of the pseudo-label perspectives is used to obtain a second joint field of view; Determine an intersecting visual field based on the first joint visual field and the second joint visual field; Determine a second ground crowd density map based on the intersecting visual field and the crowd labels based on each labeled visual angle, and determine a third ground crowd density map based on the intersecting visual field and the second predicted ground crowd density map; Constructing a second loss function term based on the second ground crowd density map and the third ground crowd density map; The preset task model is trained based on the second loss function term.

[0011] A second aspect of the present application provides a crowd positioning method, using a target task model corresponding to a crowd task trained by the above-mentioned model training method, where the crowd task is a crowd positioning task; the crowd positioning method specifically includes: Acquire a multi-view image, and input the multi-view image into the target task model; The crowd positioning result is outputted through the target task model.

[0012] The third aspect of the present application provides a model training device, wherein the model training device specifically includes: An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of multi-view images; A first selection module, configured to select a first perspective based on the training data set, and add the first perspective to a labeled perspective set; A first training module is used to annotate the training data set with a crowd based on the labeled view set to obtain an annotated training data set, and to train an initial task model corresponding to a crowd task based on the annotated training data set, wherein each labeled view in the annotated training data set carries a crowd label; The second selection module is used to select a second perspective based on the initial task model and the labeled perspective set, and add the second perspective to the labeled perspective set, and re-execute the step of annotating the training data set based on the labeled perspective set to obtain an annotated training data set until the first perspective is selected. Perspective, where is a positive integer; The second training module is used to The perspective is added to a labeled perspective set, the training data set is crowd labeled based on the labeled perspective set to obtain a labeled training data set, and a preset task model corresponding to the crowd task is trained based on the labeled training data set to obtain a target task model corresponding to the crowd task.

[0013] The fourth aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the model training methods described above.

[0014] A fifth aspect of the present application provides a terminal device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in any of the model training methods described above are implemented.

[0015] Beneficial effect: Compared with the prior art, the present application provides a model training method, a crowd positioning method, an apparatus, a device and a medium, wherein the method comprises selecting a first perspective based on the training data set, and adding the first perspective to a labeled perspective set; annotating the training data set with a crowd based on the labeled perspective set to obtain a labeled training data set, and training an initial task model corresponding to a crowd task based on the labeled training data set; selecting a second perspective based on the initial task model and the labeled perspective set, and adding the second perspective to the labeled perspective set, and re-executing the step of annotating the training data set with a crowd based on the labeled perspective set to obtain a labeled training data set, until the first perspective is selected. perspective; The perspective is added to the labeled perspective set, the training data set is annotated with people based on the labeled perspective set to obtain an annotated training data set, and a preset task model corresponding to the crowd task is trained based on the annotated training data set to obtain a target task model corresponding to the crowd task. In the perspective selection process, the application fully considers the prediction results of the initial task model, so that the selected perspective can cover more people in the scene, thereby reducing the situation where people in the scene are omitted, thereby improving the detection accuracy of the trained target task model, and thereby improving the prediction accuracy of multi-perspective tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of the model training method provided in an embodiment of the present application.

[0018] Figure 2 A flow chart of the principles of the model training method provided in an embodiment of the present application.

[0019] Figure 3 Schematic flow chart of the principle of training the initial task model based on the labeled training dataset.

[0020] Figure 4 Schematic diagram of scene coverage and ground crowd density.

[0021] Figure 5 Principle flow chart for training a preset task model based on a labeled training dataset.

[0022] Figure 6 This is a functional block diagram of the model training device provided in an embodiment of the present application.

[0023] Figure 7 A functional block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application provide a model training method, a crowd positioning method, an apparatus, a device and a medium. To make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0026] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0027] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0028] After research, it was found that multi-view crowd tasks (such as multi-view crowd counting and multi-view crowd positioning) solve the large-scale scene coverage, serious occlusion and ambiguity problems encountered in single-view tasks by fusing multiple camera perspectives that are calibrated at the same time. However, existing multi-view crowd tasks mainly focus on task models and generally use randomly sampled multi-view inputs. The randomly sampled multi-view method only focuses on the crowd prediction results of the input perspective, but not the scene-level crowd prediction results. This may lead to inaccurate scene-level crowd predictions due to the large number of scene-level people being omitted. In addition, a large amount of labeled data is used each time the task model is trained, which requires a high labeling cost.

[0029] In order to solve the above problem, in an embodiment of the present application, a first perspective is selected based on the training data set, and the first perspective is added to the labeled perspective set; the training data set is annotated with a crowd based on the labeled perspective set to obtain a labeled training data set, and an initial task model corresponding to the crowd task is trained based on the annotated training data set; a second perspective is selected based on the initial task model and the labeled perspective set, and the second perspective is added to the labeled perspective set, and the step of annotating the training data set with a crowd based on the labeled perspective set to obtain a labeled training data set is re-executed until the first perspective is selected. perspective; The perspective is added to the labeled perspective set, the training data set is annotated with people based on the labeled perspective set to obtain an annotated training data set, and a preset task model corresponding to the crowd task is trained based on the annotated training data set to obtain a target task model corresponding to the crowd task. In the perspective selection process, the application fully considers the prediction results of the initial task model, so that the selected perspective can cover more people in the scene, thereby reducing the situation where people in the scene are omitted, thereby improving the detection accuracy of the trained target task model, and thereby improving the prediction accuracy of multi-perspective tasks.

[0030] At the same time, this application is in the process of selecting perspectives and selecting Pseudo-label perspectives will be introduced in the model training process after the perspective. By indirectly using the crowd labels of the labeled perspective as the crowd labels of the pseudo-label perspective, the task model can be trained with a large amount of unlabeled data under limited labels, thereby improving the performance of the model and reducing the label cost.

[0031] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0032] This embodiment provides a model training method, such as Figure 1 and Figure 2 As shown, the method includes: S10. Acquire a training data set, where the training data set includes a plurality of multi-view images.

[0033] Specifically, the training data set includes a number of multi-view images, which can be obtained by shooting the same scene or multiple scenes, wherein the multi-view images corresponding to the same scene have different shooting times, and the multi-view images corresponding to different scenes may have multi-time images with the same shooting time. For example, multiple calibrated camera perspectives can be deployed in a scene, and the scene can be shot at different times by the multiple calibrated camera perspectives to obtain a number of multi-view images; or, multiple calibrated camera perspectives can be deployed in multiple scenes, and then each scene can be shot separately to obtain a number of multi-view images.

[0034] It should be noted that when the training data set corresponds to multiple scenes, Then select the viewpoint in all scenes. After the multi-view images corresponding to all scenes are trained, the target task model is trained. The selection process of each perspective and the processing process of each frame of multi-view image in the training process of the target task model are the same. Based on this, for the sake of convenience, here we take the training data set corresponding to a scene as an example, and The process of selecting a perspective and the training process of the target task model are explained in this paper.

[0035] Furthermore, after the multi-view images are captured, the data set consisting of the captured multi-view images can be directly used as the training data set, or the captured multi-view images can be first screened, and then the data set consisting of the screened multi-view images can be used as the training data set. In an embodiment of the present application, in order to improve the image quality of the multi-view images included in the training data set, after the multi-view images are captured, the captured multi-view images will be first screened, and then the data set consisting of the screened multi-view images will be used as the training data set to further reduce the label cost. Among them, the process of screening the captured multi-view images can be: Obtaining the maximum viewing angle with the largest field of view in the scene corresponding to the captured image set; Counting the crowd counting results in each frame of the multi-view image with the maximum viewing angle in the captured image set, and taking the multi-view image with the largest crowd counting result as the first frame of the multi-view image; Add the first frame of the multi-view image to the selected image set, and remove the first frame of the multi-view image from the captured image set; Calculate the cosine similarity of each frame of the multi-view image in the selected image set and each frame of the multi-view image in the captured image set at the maximum viewing angle, and select the second frame of the multi-view image based on the calculated cosine similarity; Add the second multi-view image to the selected image set, and remove the second multi-view image from the captured image set; Calculate the cosine similarity of each frame of the multi-view image in the selected image set and each frame of the multi-view image in the captured image set at the maximum viewing angle, and select the third frame of the multi-view image based on the calculated cosine similarity; And so on until you select Frame multi-view image, the first The frame multi-view image is added to the selected image set, and the selected image set is used as the training data set of the scene corresponding to the captured image set, where: Is a positive integer.

[0036] Specifically, the crowd counting result in the multi-view image can be obtained by using a preset single-view counting model (for example, a DM-Count model, etc.), that is, the crowd count of the single-view image corresponding to the maximum view is counted by the single-view counting model, and the crowd count is used as the crowd counting result in the multi-view image. After obtaining the crowd counting result corresponding to each frame of the multi-view image, the multi-view image with the largest crowd counting result can be used as the first frame of the multi-view image.

[0037] Further, when selecting a multi-view image based on the calculated cosine similarity, when the selected image set has only one frame of multi-view image, the multi-view image with the smallest calculated cosine similarity can be directly used as the second multi-view image, that is, the multi-view image with the smallest cosine similarity with the first frame of multi-view image at the maximum viewing angle in the captured image set is selected as the second multi-view image. When the selected image set includes multiple frames of multi-view images, after all cosine similarities are calculated, for each frame of multi-view image in the captured image set, the sum of its cosine similarities with each frame of multi-view image in the selected image set is calculated, and then the multi-view image corresponding to the smallest sum of cosine similarities among the sums of calculated cosine similarities is used as the selected multi-view image. For example, when the selected image set includes the first frame of multi-view image and the second frame of multi-view image, the multi-view image with the smallest sum of cosine similarities with the first frame of multi-view image and the second frame of multi-view image can be selected as the third multi-view image.

[0038] It should be noted that when multi-view images are taken of multiple scenes, the shooting data set obtained by shooting each scene can be screened for multi-view images according to the above process, and then the set of multi-view images screened for each scene can be used as a training data set.

[0039] This application uses the shooting data set corresponding to the scene to select If we use multi-view images as training data, we only need to Frame multi-view images are annotated, so that as little training data as possible can be selected while meeting the training requirements, and the training data that needs to be annotated can be calculated, thereby reducing the crowd labeling cost.

[0040] S20. Select a first perspective based on the training data set, and add the first perspective to a labeled perspective set.

[0041] Specifically, the first perspective is a perspective in a multi-perspective image, wherein the first perspective is determined based on the crowd counting result of each perspective in the multi-perspective image on the training data set, and the determination process can be: first obtain the crowd counting result of each perspective in each frame of the multi-perspective image on each frame of the multi-perspective image in the training data set, and then sum the crowd counting results of each perspective on the selected F-frame multi-perspective image to obtain the sum of the crowd counting results corresponding to each perspective, and finally select the perspective with the largest sum of the crowd counting results as the first perspective. Of course, in practical applications, other methods can also be used to select the first perspective, such as randomly selecting a perspective as the first perspective, or randomly selecting a perspective as the first perspective from the perspectives whose sum of the crowd counting results is greater than the preset crowd counting number, etc.

[0042] In the embodiment of the present application, by selecting the perspective with the largest sum of crowd counting results as the first perspective, the first perspective can cover more people in the scene, and in the subsequent perspective selection, the initial task model obtained by training the first perspective is selected, so that the subsequent selected perspective can also cover more people in the scene, so that the selected perspective The viewing angle can cover more people in the scene.

[0043] S30, annotating the training data set with a crowd based on the labeled viewpoint set to obtain an annotated training data set, and training an initial task model corresponding to the crowd task based on the annotated training data set.

[0044] Specifically, the labeled perspective set includes actively selected perspectives, and crowd labeling of the training data set based on the labeled perspective set refers to crowd labeling of single-perspective images corresponding to the labeled perspectives in the labeled perspective set. For example, when the labeled perspective set includes the first perspective, the single-perspective image corresponding to the first perspective is labeled. Based on this, the multi-perspective images included in the labeled training data set are the same as the multi-perspective images included in the training data set, and the single-perspective images corresponding to the labeled perspectives in the labeled training data set carry crowd labels, while the single-perspective images of all perspectives in the training data set do not carry crowd labels.

[0045] When training the initial task model corresponding to the crowd task based on the labeled training data set, the single-view image corresponding to the labeled perspective in the labeled perspective set can be directly used to train the initial task model corresponding to the crowd task. For example, the single-view image corresponding to the labeled perspective is input into the initial task model corresponding to the crowd task, and the predicted ground crowd density map is output by the initial task model corresponding to the crowd task. Then, a loss function term is constructed based on the predicted ground crowd density map and crowd labels corresponding to each single-view image, and the initial task model corresponding to the crowd task is trained based on the loss function term, wherein the crowd task can be a crowd positioning task, a crowd counting task, etc.

[0046] Furthermore, the labeled perspective with crowd labels and the perspective without crowd labels are located in the same scene, and the crowd labels of the labeled perspectives can be used as pseudo crowd labels for the perspectives without crowd labels, so that the single-view images of the perspectives without crowd labels can be added to the training of the model during the training process. Based on this, in one implementation, the initial task model corresponding to the crowd training task based on the labeled training data set specifically includes: For each multi-view image frame in the training data set, a pseudo-label perspective is selected for the multi-view image; The initial task model corresponding to the crowd task is trained based on the single-view images corresponding to each labeled view and pseudo-labeled view.

[0047] Specifically, the pseudo-label perspective is an unlabeled perspective in the multi-view image, that is, the pseudo-label perspective is included in the perspective set corresponding to the multi-view image, but not in the labeled perspective set. Among them, the pseudo-label perspectives corresponding to each frame of the multi-view image in the training data set can be the same; it can also be that the pseudo-label perspectives corresponding to some multi-view images are the same, and the pseudo-label perspectives corresponding to some multi-view images are different; it can also be that the pseudo-label perspectives corresponding to each frame of the multi-view image are different. In an embodiment of the present application, an unlabeled perspective can be randomly selected from the unlabeled perspectives corresponding to each frame of the multi-view image, and the randomly selected unlabeled perspective is used as the pseudo-label perspective corresponding to the multi-view image. Of course, in practical applications, other methods can also be used, for example, randomly selecting a frame of multi-view image in the training data set, and then randomly selecting an unlabeled perspective from the unlabeled perspectives corresponding to the multi-view image, and using the selected unlabeled perspective as the pseudo-label perspective corresponding to each frame of the multi-view image.

[0048] The embodiment of the present application selects a pseudo-label perspective for each frame of the multi-view image, so that the initial task model corresponding to the single-view image training task based on each labeled perspective in the labeled perspective set can be converted into an initial task model corresponding to the single-view image training task based on each labeled perspective in the labeled perspective set and the single-view image training task corresponding to the pseudo-label perspective (e.g., the labeled perspective set includes perspectives, then the model will be used during training The perspective is converted to use This can improve the training speed of the initial task model. Of course, in practical applications, multiple pseudo-label perspectives can also be selected, for example, 2, 3, etc.

[0049] Exemplarily, the initial task model corresponding to the task of training a crowd based on the added labeled training data set specifically includes: Inputting the single-view images corresponding to each labeled view and pseudo-labeled view into an initial task model corresponding to the crowd task, and outputting a first predicted ground crowd density map through the initial task model; Obtaining a union of the visual fields of the labeled perspectives in the labeled perspective set to obtain a first joint visual field; Determine a first ground crowd density map based on the first predicted ground crowd density map and the first joint field of view; Constructing a first loss function term based on the first ground crowd density map and the crowd labels of the labeled viewpoints; The initial task model is trained based on the first loss function term.

[0050] Specifically, Figure 3 As shown, the visual field of each labeled perspective is obtained, and then all the visual fields are taken as a union to obtain a first joint visual field. After obtaining the first joint visual field, the first predicted ground crowd density map is masked by the first joint visual field to obtain the first ground crowd density map. For example, the first predicted ground crowd density map is point-multiplied with the first joint visual field to obtain the first ground crowd density map, so that the overlapping area of ​​the first predicted ground crowd density map with the first joint visual field remains unchanged, and the points in other areas become preset values ​​(such as 0, etc.).

[0051] After obtaining the first ground crowd density map, a first loss function term is constructed based on the first ground crowd density map and the crowd labels of the labeled viewpoints, and then the initial task model is trained based on the first loss function term until the training process of the initial task model reaches a preset training index, wherein the preset training index may be that the number of training times reaches a preset number of times, or that the model accuracy of the trained initial task model reaches a preset requirement, etc. In addition, it should be noted that the construction method of the first loss function and the optimization process of the model parameters of the initial task model based on the first loss function can both adopt the process of optimizing the model parameters using the loss function in the training process of the existing crowd counting model and the crowd positioning model. At the same time, when training the initial task model, the multi-view images in the training data set can also be divided into several training batches, each training batch includes multi-view images of a certain data, and each multi-view image determines the first ground crowd density map according to the above process, and then the first loss function is constructed based on all the multi-view images in the training batch, etc. For example, when the crowd task is a multi-view crowd counting task, the initial task model can use CVCS as the baseline model, and add FPN on the basis of the single-view decoder to capture the multi-scale information of the image; when the crowd task is a multi-view crowd localization task, the MVDet model can be used.

[0052] S40, selecting a second perspective based on the initial task model and the labeled perspective set, and adding the second perspective to the labeled perspective set, and re-performing the step of annotating the training data set based on the labeled perspective set to obtain an annotated training data set, until the first perspective is selected. Perspective.

[0053] Specifically, the initial task model used to select the second perspective is the initial task model trained with the labeled training data set corresponding to the first perspective, that is, the initial task model is trained based on the labeled training data set, and when the initial task model meets the preset requirements, the second perspective is selected based on the initial task model and the labeled perspective set. The second perspective is included in all perspectives corresponding to the multi-perspective image, which is different from the first perspective, that is, it is not included in the labeled perspective set.

[0054] After the second perspective is selected, the second perspective is added to the labeled perspective set, and then the crowd annotation of the training data set based on the labeled perspective set is re-executed to obtain a labeled training data set, so as to obtain an initial task model trained based on the labeled training data set corresponding to the labeled perspective set with the second perspective added, and a third perspective is selected based on the initial task model and the labeled perspective set including the first perspective and the second perspective, and so on until the third perspective is selected. Perspective, is a positive integer. The determination of the labeled training data set after each viewing angle selection and the training process of the initial task model are the same as the above process, which will not be repeated here.

[0055] further, Indicates the total number of actively selected perspectives, which can be preset, or determined according to the training requirements of the target task model corresponding to the crowd task, or determined based on the crowd coverage of the scene (for example, when selecting the first When viewing angle, obtain The ratio of the people covered by the single-view image of the perspective to the people included in the scene. If the ratio reaches the preset ratio threshold, As and stop selecting the view, etc.). In addition, 2, 3, ..., Each of The selection process of perspectives is the same, and is based on the initial task model and the labeled perspective set. The perspective selection process is explained by taking perspective as an example.

[0056] Exemplarily, the The perspective selection process specifically includes: Calculate the perspective evaluation value of each unlabeled perspective and the labeled perspective set on each frame of the multi-view image in the training data set; Calculate the perspective evaluation value and the perspective evaluation value corresponding to each unlabeled perspective, and select the first one in the unlabeled perspective based on the perspective evaluation value and Perspective.

[0057] Specifically, the perspective evaluation value reflects the probability of the unlabeled perspective being selected relative to the multi-view image, and the perspective evaluation value and the perspective evaluation value reflect the probability of the unlabeled perspective being selected relative to the training data set. When viewing angle, the viewing angle evaluation value and the maximum unlabeled viewing angle can be selected as the first Perspective. The perspective evaluation value is determined based on the scene coverage, the average perspective distance, and the perspective difference. The scene coverage is used to reflect the crowd coverage status of the perspective. The larger the scene coverage, the more people the perspective covers. Conversely, the smaller the scene coverage, the fewer people the perspective covers. The average perspective distance is used to reflect the distance relationship between the scene area and the image acquisition device (such as a camera). The average perspective distance is used to make the distance between the scene area and the image acquisition device meet the preset requirements to avoid the people in the multi-perspective image being too small, thereby ensuring the clarity of the people in the multi-perspective image. Perspective difference is used to reflect the sparsity between perspectives. The perspective difference ensures the dispersion between perspectives and avoids the selected perspectives from gathering together.

[0058] In one implementation, the process of obtaining the viewing angle evaluation value may include: For each unlabeled perspective, a single-perspective image corresponding to the unlabeled perspective and single-perspective images corresponding to each labeled perspective are used as input items, and a ground crowd density map corresponding to the unlabeled perspective is determined by the above-mentioned initial task model, and a crowd mask map corresponding to the unlabeled perspective is determined based on the ground crowd density map corresponding to the unlabeled perspective, wherein the initial task model is selected after the first After training, =2,3,..., ; Determine the scene coverage corresponding to the unlabeled perspective based on the crowd mask map, determine the average perspective distance corresponding to the unlabeled perspective based on the ground crowd density map and the crowd mask map, and calculate the perspective difference corresponding to the unlabeled perspective; A view evaluation value of the unlabeled view is determined based on the scene coverage, the average view distance, and the view difference.

[0059] Specifically, for scene coverage, the higher the scene coverage, the more people the selected perspective can cover. Therefore, the scene coverage can be used as an evaluation item for perspective selection. The scene coverage can be determined based on the field of view of the perspective and the area of ​​the scene. Specifically, let the first The field of view of the scene is ,like Figure 4 As shown, the joint visual field of all labeled views in the labeled view set is , scene The area of ​​the region is , Indicates the width of the scene area, represents the height of the scene area, then the scene coverage can be expressed as , represents the number of labeled views in the labeled view set and the number of unlabeled views. The number of labeled views in the labeled view set is .

[0060] In addition, in practical applications, due to the possible occlusion in multi-view images, the field of view of the scene cannot directly represent the actual coverage range. In order to improve the accuracy of scene coverage, the ground crowd density map predicted by the initial task model can be used to determine the scene coverage. To determine the scene coverage, specifically, the ground crowd density map used to reflect the probability distribution of the crowd in the scene can be predicted by the initial task model. , and then based on the ground population density map Crowd mask map to determine the area where people are located , for example, Figure 4 As shown in the ground population density map Applying a threshold (such as a pre-set threshold, etc.) to the crowd mask map can be obtained. Finally, the crowd mask As the joint visual field, the scene coverage is determined by calculating the ratio of the joint visual field to the area of ​​the region, wherein the calculation formula of the scene coverage can be:

[0061] in, Indicates the coverage of the scene.

[0062] For the average viewing distance, the average viewing distance can reflect the clarity of the person in the single-view image corresponding to the view, and the average viewing distance can be used as an evaluation item for view selection. The average viewing distance can be the average of the average distance values ​​of the visible points of the labeled view. Specifically, for the crowd mask map Every visible point on , by making the visible point Distance in world coordinates The visible point can be seen in the perspective The inverse of the viewing distance and the visible point The average distance value of the visible point The average distance value , Indicates visible points The average distance value of Representation perspective The position in the world coordinate system, and when the visible point Falling in perspective The visual field of the participating visible points To this end, the average viewing angle distance can be calculated by The average visual distance of all visible points in the joint visual field of the viewpoints is determined by , Represents the average viewing distance corresponding to the labeled viewing set.

[0063] In addition, due to the ground crowd density map It reflects the probability of people appearing, so that the ground crowd density map can be As a crowd mask The weight value of each point on the graph, the greater the probability, the more likely the person is to appear at that point, and a higher weight is assigned to that point. The average distance value can also be expressed as .

[0064] Regarding the perspective difference, since multi-perspective can greatly reduce the occlusion and ambiguity of objects compared to single perspective, the perspective difference reflects the sparsity between perspectives, ensuring that the perspectives are relatively dispersed and not gathered together. The perspective difference can be calculated by perspective similarity measurement, where the expression of perspective difference can be:

[0065] in, Indicates the difference in perspective, Representation perspective The direction of the optical axis in the world coordinate system, that is, the direction vector, Representation perspective The direction of the optical axis in the world coordinate system, that is, the direction vector, and are all constant terms, To avoid division by zero, Used to control the sensitivity of the view difference term.

[0066] Furthermore, after obtaining the scene coverage, average viewing distance, and viewing difference, the product of the scene coverage, average viewing distance, and viewing difference can be used as the viewing angle evaluation value, that is, the viewing angle evaluation value It can be expressed as .

[0067] In the model training process, the embodiment of the present application first performs an active perspective selection training process. Specifically, after selecting the training data set and the first perspective corresponding to each training scene, the first perspective is added to the labeled perspective set, and the initial task model is trained on the labeled perspective set and training data set selected for each training scene. When the training index of the initial task model reaches the set threshold, the perspective selection begins. After the perspective is selected, the selected perspective is added to the labeled perspective set and the labeled crowd label, and then the initial task model is retrained on the labeled perspective set and training data set selected for each training scene, and so on until the first perspective is selected. Viewing angle ( The number of pre-labeled perspectives). This application combines the perspective selection process with the model training process in the active perspective selection training process, so that the task model can guide the perspective selection process, improve the accuracy of perspective selection, and also ensure that the selected perspective can ensure the accuracy of the task model prediction results. In addition, in the process of guiding perspective selection through the task model, the perspective evaluation value determined based on scene coverage, average perspective distance and perspective difference is used as the basis for perspective selection, which fully considers the geometric information of the scene and perspective, the position information of the crowd and the probability distribution information of the crowd, and can select perspectives covering more scenes. The crowd can be better used for model training and model testing, and better scene-level crowd prediction results are obtained.

[0068] S50, The perspective is added to a labeled perspective set, the training data set is crowd labeled based on the labeled perspective set to obtain a labeled training data set, and a preset task model corresponding to the crowd task is trained based on the labeled training data set to obtain a target task model corresponding to the crowd task.

[0069] Specifically, in After the perspective is selected, the perspective selection process is completed, and only the training process is required. After a perspective is added to the set of labeled perspectives, it is The perspective labels the crowd and marks The training data set from each perspective is used as the final labeled training data set, and the preset task model corresponding to the crowd task is trained based on the labeled training data set. The initial task model after training the annotated training data set of the perspective can also be a preset default task model. The initial task model trained with the labeled training data set of the perspective is used as the preset task model, and then the preset task model is trained based on the labeled training data set to obtain the target task model.

[0070] Furthermore, when training the preset task model corresponding to the crowd task based on the labeled training data set, it is possible to directly The single-view image corresponding to the perspective is used as the input of the preset task model, and the predicted ground crowd density map is then determined by the preset task model. Then, the loss function term is constructed based on the predicted ground crowd density map and the crowd label. The model parameters of the preset task model are optimized through the loss function term to obtain the target task model.

[0071] However, in practical applications, due to The perspective is only part of the perspective in the multi-view image, and the multi-view image also has other unlabeled perspectives. In the post-training process of each view, the single-view image corresponding to the unlabeled view can be included in the training process of the preset task model, and training labels are generated for the input items including the unlabeled view to improve the model performance and generalization of the target training model. Specifically, considering that the labeled view and the unlabeled view are both in the same scene, the crowd labels of the labeled view can be used to construct pseudo crowd labels for the unlabeled view.

[0072] Based on this, the preset task model corresponding to the crowd task is trained based on the labeled training data set to obtain the target task model corresponding to the crowd task specifically includes: For each multi-view image in the training dataset, select target perspectives, selected from the unlabeled perspectives other than the labeled perspective set pseudo-label perspectives, where , is a positive integer; Based on the Target perspective and The preset task model corresponding to the crowd task is trained using single-view images corresponding to the pseudo-labeled view.

[0073] Specifically, the target perspective can be a labeled perspective in the labeled perspective set. For example, the target perspective can be randomly selected from the labeled perspective set. Tagged perspectives as Target perspective. After the target perspective, select the unlabeled perspective pseudo-labeled perspectives, so that the number of single-view images used to train the preset task model is Then Target perspective and The single-view image corresponding to the pseudo-label perspective is used as the input item of the preset task model. Target perspective and The preset task model is trained based on the single-view images corresponding to the pseudo-labeled viewpoints.

[0074] In one implementation, if Figure 5 As shown, the preset task model corresponding to the task of training a crowd based on the added labeled training data set specifically includes: Will Target perspective and The single-view image corresponding to the pseudo-label perspective is input into the preset task model, and the second predicted ground crowd density map is output through the preset task model; Obtain the union of the visual fields of each labeled perspective to obtain a first joint visual field, and Target perspective and The union of the fields of view of the pseudo-label perspectives is used to obtain a second joint field of view; Determine an intersecting visual field based on the first joint visual field and the second joint visual field; Determine a second ground crowd density map based on the intersecting visual field and the crowd labels based on each labeled visual angle, and determine a third ground crowd density map based on the intersecting visual field and the second predicted ground crowd density map; Constructing a second loss function term based on the second ground crowd density map and the third ground crowd density map; The preset task model is trained based on the second loss function term.

[0075] Specifically, the process of determining the first joint visual field and the second joint visual field is the same as the process of determining the first joint visual field mentioned above, and will not be repeated here. The intersecting visual field is the intersection area of ​​the first joint visual field and the second joint visual field, that is, the overlapping area of ​​the first joint visual field and the second joint visual field can be obtained, and the obtained overlapping area is used as the intersecting visual field. In addition, the second ground crowd density map, the third ground crowd density map, the second loss function term, and the training of the preset task model based on the second loss function terms are the same as those in the above-mentioned perspective selection process, and will not be repeated here.

[0076] The embodiment of the present application adds unlabeled perspectives in both the active perspective selection training process and the separate model training process, and uses labeled perspectives to construct pseudo-crowd labels for input items with unlabeled perspectives, thereby achieving the use of a small number of labeled perspectives to provide labels for a large number of unlabeled perspectives, allowing a large number of unlabeled perspectives to be used for model training, reducing labeling costs.

[0077] In addition, after the target task model is trained, it will be tested. Specifically, Figure 2 As shown in the figure, the test process includes an active perspective selection test process and a test label generation process, wherein the active perspective selection test process is the same as the above-mentioned active perspective selection training process, and in the test label generation, the test index takes into account all people in the scene (ie, scene-level labels), instead of only including people appearing in the selected perspective. This can not only reflect the quality of the selected perspective, but also reflect the quality of the model training, and in the perspective selection process, all frames of the test scene are used for perspective selection.

[0078] In summary, this embodiment provides a model training method, which includes selecting a first perspective based on the training data set, and adding the first perspective to a labeled perspective set; annotating the training data set with a crowd based on the labeled perspective set to obtain a labeled training data set, and training an initial task model corresponding to a crowd task based on the labeled training data set; selecting a second perspective based on the initial task model and the labeled perspective set, and adding the second perspective to the labeled perspective set, and re-executing the step of annotating the training data set with a crowd based on the labeled perspective set to obtain a labeled training data set, until the first perspective is selected. perspective; The perspective is added to the labeled perspective set, the training data set is annotated with people based on the labeled perspective set to obtain an annotated training data set, and a preset task model corresponding to the crowd task is trained based on the annotated training data set to obtain a target task model corresponding to the crowd task. In the perspective selection process, the application fully considers the prediction results of the initial task model, so that the selected perspective can cover more people in the scene, thereby reducing the situation where people in the scene are omitted, thereby improving the detection accuracy of the trained target task model, and thereby improving the prediction accuracy of multi-perspective tasks.

[0079] In order to further illustrate the better generalization and scene-level crowd prediction results achieved by combining active perspective selection with model training in the model training method provided in the embodiment of the present application. The embodiment of the present application uses the crowd tasks of multi-perspective crowd counting task and multi-perspective crowd positioning task as examples for comparative experiments. In the multi-perspective crowd counting task, the results of several methods with different training methods, different perspective selection methods, and the addition of pseudo-label perspectives under the same benchmark model are compared, as shown in Table 1, which proves that the method provided in the embodiment of the present application can select better perspectives for model training and testing, and obtain better scene-level crowd prediction results. It also proves that the method provided in the embodiment of the present application can use a large amount of unlabeled data for training while using only a small amount of labeled data, thereby enhancing the generalization and performance of the model. In the multi-view crowd positioning task, the results of using different view selection methods and pseudo-label view under the same model are compared, as shown in Tables 2 and 3. The multi-view selection method (MVSelect) uses all label data, which proves that the method provided in the embodiment of the present application can achieve good results using only a small amount of label data, which reduces the label cost. It also proves the effectiveness of the active view selection framework of the method provided in the embodiment of the present application that takes into account the scene and view geometry information, crowd location information, and crowd probability distribution information.

[0080] Table 1 Schematic diagram of the results of the multi-view crowd counting task CVCS dataset

[0081] Table 2 Schematic diagram of the results of the MultiviewX dataset for the multi-view crowd localization task

[0082] Table 3 Schematic diagram of the results of the Wildtrack dataset for the multi-view crowd localization task

[0083] Based on the above model training method, this embodiment provides a crowd positioning method, using a target task model corresponding to a crowd task trained by the above model training method, where the crowd task is a crowd positioning task; the crowd positioning method specifically includes: Acquire a multi-view image, and input the multi-view image into the target task model; The crowd positioning result is outputted through the target task model.

[0084] Based on the above model training method, this embodiment provides a model training device, such as Figure 6 As shown, the model training device specifically includes: An acquisition module 100 is used to acquire a training data set, wherein the training data set includes a plurality of multi-view images; A first selection module 200, configured to select a first perspective based on the training data set, and add the first perspective to a labeled perspective set; A first training module 300 is used to annotate the training data set with a crowd based on the labeled view set to obtain an annotated training data set, and to train an initial task model corresponding to a crowd task based on the annotated training data set, wherein each labeled view in the annotated training data set carries a crowd label; The second selection module 400 is used to select a second perspective based on the initial task model and the labeled perspective set, and add the second perspective to the labeled perspective set, and re-execute the step of annotating the training data set based on the labeled perspective set to obtain an annotated training data set until the first perspective is selected. Perspective, where is a positive integer; The second training module 500 is used to The perspective is added to a labeled perspective set, the training data set is crowd labeled based on the labeled perspective set to obtain a labeled training data set, and a preset task model corresponding to the crowd task is trained based on the labeled training data set to obtain a target task model corresponding to the crowd task.

[0085] Based on the above-mentioned model training method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the model training method described in the above-mentioned embodiment.

[0086] Based on the above model training method, the present application also provides a terminal device, such as Figure 7 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communications interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communications interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0087] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0088] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0089] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0090] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The model training method specifically includes: Acquire a training data set, wherein the training data set includes a plurality of multi-view images; Selecting a first perspective based on the training data set, and adding the first perspective to a labeled perspective set; Based on the labeled view set, the training data set is labeled with a crowd to obtain a labeled training data set, and an initial task model corresponding to a crowd task is trained based on the labeled training data set, wherein each labeled view in the labeled training data set carries a crowd label; Based on the initial task model and the labeled perspective set, a second perspective is selected, and the second perspective is added to the labeled perspective set, and the step of annotating the training data set based on the labeled perspective set to obtain an annotated training data set is re-executed until the first perspective is selected. Perspective, where is a positive integer; The first The perspective is added to a labeled perspective set, the training data set is crowd labeled based on the labeled perspective set to obtain a labeled training data set, and a preset task model corresponding to the crowd task is trained based on the labeled training data set to obtain a target task model corresponding to the crowd task.

2. The model training method according to claim 1, characterized in that: The said The perspective selection process specifically includes: Calculating a perspective evaluation value of each unlabeled perspective and a labeled perspective set on each frame of the multi-perspective image in the training data set, wherein the perspective evaluation value is determined based on scene coverage, average perspective distance, and perspective difference; Calculate the perspective evaluation value and the perspective evaluation value corresponding to each unlabeled perspective, and select the first one in the unlabeled perspective based on the perspective evaluation value and Perspective.

3. The model training method according to claim 1, characterized in that: The initial task model corresponding to the task of training a group of people based on the labeled training data set specifically includes: For each multi-view image frame in the training data set, a pseudo-label perspective is selected for the multi-view image; The initial task model corresponding to the crowd task is trained based on the single-view images corresponding to each labeled view and pseudo-labeled view.

4. The model training method according to claim 3, characterized in that: The initial task model corresponding to the task of training a crowd based on single-view images corresponding to each labeled view and pseudo-label view specifically includes: Inputting the single-view images corresponding to each labeled view and pseudo-labeled view into an initial task model corresponding to the crowd task, and outputting a first predicted ground crowd density map through the initial task model; Obtaining a union of the visual fields of the labeled perspectives in the labeled perspective set to obtain a first joint visual field; Determine a first ground crowd density map based on the first predicted ground crowd density map and the first joint field of view; Constructing a first loss function term based on the first ground crowd density map and the crowd labels of the labeled viewpoints; The initial task model is trained based on the first loss function term.

5. The model training method according to claim 1, characterized in that: The step of training a preset task model corresponding to a crowd task based on the labeled training data set to obtain a target task model corresponding to the crowd task specifically includes: For each multi-view image in the training dataset, select target perspectives, selected from the unlabeled perspectives other than the labeled perspective set pseudo-label perspectives, where , is a positive integer; Based on the Target perspective and The preset task model corresponding to the crowd task is trained using single-view images corresponding to the pseudo-labeled view.

6. The model training method according to claim 5, characterized in that: Based on the Target perspective and The preset task models corresponding to the single-view image training crowd task corresponding to the pseudo-label perspective include: Will Target perspective and The single-view image corresponding to the pseudo-label perspective is input into the preset task model, and the second predicted ground crowd density map is output through the preset task model; Obtain the union of the visual fields of each labeled perspective to obtain a first joint visual field, and Target perspective and The union of the fields of view of the pseudo-label perspectives is used to obtain a second joint field of view; Determine an intersecting visual field based on the first joint visual field and the second joint visual field; Determine a second ground crowd density map based on the intersecting visual field and the crowd labels based on each labeled visual angle, and determine a third ground crowd density map based on the intersecting visual field and the second predicted ground crowd density map; Constructing a second loss function term based on the second ground crowd density map and the third ground crowd density map; The preset task model is trained based on the second loss function term.

7. A crowd positioning method, characterized in that: Using a target task model corresponding to a crowd task obtained by training using the model training method according to any one of claims 1 to 6, the crowd task is a crowd positioning task; the crowd positioning method specifically comprises: Acquire a multi-view image, and input the multi-view image into the target task model; The crowd positioning result is outputted through the target task model.

8. A model training device, characterized in that: The model training device specifically comprises: An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of multi-view images; A first selection module, configured to select a first perspective based on the training data set, and add the first perspective to a labeled perspective set; A first training module is used to annotate the training data set with a crowd based on the labeled view set to obtain an annotated training data set, and to train an initial task model corresponding to a crowd task based on the annotated training data set, wherein each labeled view in the annotated training data set carries a crowd label; The second selection module is used to select a second perspective based on the initial task model and the labeled perspective set, and add the second perspective to the labeled perspective set, and re-execute the step of annotating the training data set based on the labeled perspective set to obtain an annotated training data set until the first perspective is selected. Perspective, where is a positive integer; The second training module is used to The perspective is added to a labeled perspective set, the training data set is crowd labeled based on the labeled perspective set to obtain a labeled training data set, and a preset task model corresponding to the crowd task is trained based on the labeled training data set to obtain a target task model corresponding to the crowd task.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the model training method as described in any one of claims 1-6.

10. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the model training method as described in any one of claims 1-6 are implemented.

Images