Method, device, equipment and storage medium for determining sample image

Abstract

The embodiments of the present application disclose a method, device, equipment and storage medium for determining a sample image, which belongs to the field of data processing. The method includes: performing false alarm analysis on a video stream to determine a first image and a false alarm result corresponding to the first image, wherein the first image is an image in the video stream where a false alarm object exists, and the false alarm object is a target object that is missed in the analysis or a target object that is erroneously analyzed. Based on the false alarm result corresponding to the first image, a plurality of foreground images and a plurality of background images are obtained from the video stream. The plurality of foreground images and the plurality of background images are fused to obtain a plurality of second images. The first image and the plurality of second images are determined as sample images. The embodiments of the present application train a first neural network model through these generated sample images, which can improve the training effect of the first neural network model, thereby effectively improving the analysis performance of the first neural network model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular to a method, apparatus, device and storage medium for determining a sample image. Background Art

[0002] In the process of reasoning about an image, the neural network model needs to analyze the target objects in the image and continuously update the model itself. As the neural network model is updated, it may forget certain types of target objects that it has learned before. When it is necessary to analyze these types of target objects again, the analysis accuracy may be low. Therefore, it is necessary to re-determine the sample images for the target objects that are easy to forget, so as to facilitate the re-training of the neural network model in the later stage and improve the analysis accuracy of the neural network model.

[0003] Related art proposes a method for determining sample images, in which a large number of images can be obtained, and images with higher uncertainty can be screened out from these images through a neural network model. Then, these images are divided according to their feature distribution characteristics, and then some images are selected as sample images according to the feature distribution characteristics. However, the use of uncertainty and feature distribution characteristics to determine sample images results in a limited number of sample images, and thus the performance improvement of the neural network model is also relatively limited. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment and storage medium for determining a sample image, which can solve the problem that the related technology has limited performance improvement in neural network models. The technical solution is as follows:

[0005] In one aspect, a method for determining a sample image is provided, the method comprising:

[0006] Performing omission and false alarm analysis on the video stream to determine a first image and omission and false alarm results corresponding to the first image, wherein the first image is an image in the video stream where an omission and false alarm object exists, and the omission and false alarm object is a target object that is omission-analyzed or a target object that is erroneously analyzed;

[0007] Based on the false alarm results corresponding to the first image, acquiring a plurality of foreground images and a plurality of background images from the video stream;

[0008] fusing the plurality of foreground images and the plurality of background images to obtain a plurality of second images;

[0009] The first image and the plurality of second images are determined as sample images.

[0010] Optionally, performing omission and false alarm analysis on the video stream to determine the first image and the omission and false alarm results corresponding to the first image includes:

[0011] By using a plurality of first neural network models and a second neural network model, a false alarm and omission analysis is performed on any frame image in the video stream to determine whether there is a false alarm and omission object in any frame image, wherein the second neural network model is a model capable of analyzing all target objects in any frame image;

[0012] If there is a false alarm object in any of the frame images, the any frame image is used as the first image, and the false alarm result corresponding to the first image is determined.

[0013] Optionally, performing missed and false alarm analysis on any frame image in the video stream by using multiple first neural network models and second neural network models to determine whether there is a missed and false alarm object in any frame image includes:

[0014] Determine the first analysis results corresponding to any one of the frames of image respectively through the multiple first neural network models to obtain multiple first analysis results, wherein the first analysis results include position information of at least one target object in any one of the frames of image and a first label of each target object;

[0015] Determine a second analysis result corresponding to any one frame of image by using the second neural network model, wherein the second analysis result includes location information of at least one target object in any one frame of image and a second label of each target object;

[0016] Based on the location information and the first label of the target object included in the plurality of first analysis results, and the location information and the second label of the target object included in the second analysis result, it is determined whether there is a missed or falsely reported object in any frame of image.

[0017] Optionally, determining whether there is a missed or falsely reported object in any frame image based on the position information and the first label of the target object included in the multiple first analysis results, and the position information and the second label of the target object included in the second analysis result, includes:

[0018] If the location information and the first label of the target object included in the multiple first analysis results are the same, and are the same as the location information and the second label of the target object included in the second analysis result, it is determined that there is no missed false alarm object in any of the frame images; otherwise, it is determined that there is a missed false alarm object in any of the frame images.

[0019] Optionally, the missed false alarm result includes location information of at least one missed false alarm object in the first image, and acquisition time of the first image;

[0020] The acquiring a plurality of foreground images and a plurality of background images from the video stream based on the false alarm result corresponding to the first image includes:

[0021] Determine at least one spatial range based on the location information of the at least one missed or falsely reported object, the at least one spatial range corresponding to the at least one missed or falsely reported object one by one;

[0022] determining a time range based on the acquisition time of the first image;

[0023] Based on the time range and the at least one spatial range, the multiple foreground images and the multiple background images are obtained from the video stream.

[0024] Optionally, acquiring the multiple foreground images and the multiple background images from the video stream based on the time range and the at least one spatial range includes:

[0025] Acquire from the video stream an image region whose acquisition time is within the time range, whose position is within the at least one space range and which is a foreground, so as to obtain a plurality of image regions;

[0026] Acquire from the video stream an image whose acquisition time is within the time range and does not have a foreground, so as to obtain a plurality of third images;

[0027] Clustering the plurality of image regions to obtain a plurality of first clustering results, and clustering the plurality of third images to obtain a plurality of second clustering results;

[0028] A plurality of image regions are selected from the plurality of first clustering results as the plurality of foreground images, and a plurality of third images are selected from the plurality of second clustering results as the plurality of background images.

[0029] Optionally, fusing the multiple foreground images with the multiple background images includes:

[0030] Determine the acquisition time of a frame of image where the multiple foreground images are located in the video stream, and the positions of the multiple foreground images in the frame of image where the multiple foreground images are located, so as to obtain the acquisition time and image positions corresponding to the multiple foreground images respectively;

[0031] Performing semantic segmentation on the multiple background images respectively to determine semantic segmentation information corresponding to the multiple background images respectively;

[0032] Determine the acquisition time of a frame of image where the multiple background images are located in the video stream to obtain the acquisition time corresponding to the multiple background images respectively;

[0033] The multiple foreground images and the multiple background images are fused based on the acquisition time and image positions respectively corresponding to the multiple foreground images and the acquisition time and semantic segmentation information respectively corresponding to the multiple background images.

[0034] In another aspect, a device for determining a sample image is provided, the device comprising:

[0035] A first determination module is used to perform a missed false alarm analysis on the video stream to determine a missed false alarm result corresponding to a first image and the first image, wherein the first image is an image in the video stream where a missed false alarm object exists, and the missed false alarm object is a target object that is missed in the analysis or a target object that is erroneously analyzed;

[0036] An acquisition module, configured to acquire a plurality of foreground images and a plurality of background images from the video stream based on the false alarm and omission results corresponding to the first image;

[0037] A fusion module, used for fusing the multiple foreground images and the multiple background images to obtain multiple second images;

[0038] The second determining module is configured to determine the first image and the plurality of second images as sample images.

[0039] Optionally, the first determining module includes:

[0040] A first determination submodule is used to perform a false alarm and omission analysis on any frame image in the video stream through a plurality of first neural network models and a second neural network model to determine whether there is a false alarm and omission object in any frame image, wherein the second neural network model is a model capable of analyzing all target objects in any frame image;

[0041] The second determination submodule is used to, if there is a missed or false alarm object in any frame of image, use the any frame of image as the first image and determine the missed or false alarm result corresponding to the first image.

[0042] Optionally, the first determining submodule includes:

[0043] A first determination unit is used to determine the first analysis results corresponding to any one of the frame images respectively through the multiple first neural network models to obtain multiple first analysis results, wherein the first analysis results include position information of at least one target object in any one of the frame images and a first label of each target object;

[0044] A second determining unit, configured to determine a second analysis result corresponding to any one frame of image by using the second neural network model, wherein the second analysis result includes position information of at least one target object in any one frame of image and a second label of each target object;

[0045] The third determination unit is used to determine whether there is a missed or falsely reported object in any frame image based on the position information and the first label of the target object included in the multiple first analysis results, and the position information and the second label of the target object included in the second analysis result.

[0046] Optionally, the third determining unit is specifically configured to:

[0047] If the location information and the first label of the target object included in the multiple first analysis results are the same, and are the same as the location information and the second label of the target object included in the second analysis result, it is determined that there is no missed false alarm object in any of the frame images; otherwise, it is determined that there is a missed false alarm object in any of the frame images.

[0048] Optionally, the missed false alarm result includes location information of at least one missed false alarm object in the first image, and acquisition time of the first image;

[0049] The acquisition module comprises:

[0050] A third determination submodule is used to determine at least one spatial range based on the location information of the at least one missed or falsely reported object, wherein the at least one spatial range corresponds to the at least one missed or falsely reported object in a one-to-one manner;

[0051] a fourth determination submodule, configured to determine a time range based on the acquisition time of the first image;

[0052] The acquisition submodule is used to acquire the multiple foreground images and the multiple background images from the video stream based on the time range and the at least one spatial range.

[0053] Optionally, the acquisition submodule is specifically used for:

[0054] Acquire from the video stream an image region whose acquisition time is within the time range, whose position is within the at least one space range and which is a foreground, so as to obtain a plurality of image regions;

[0055] Acquire from the video stream an image whose acquisition time is within the time range and does not have a foreground, so as to obtain a plurality of third images;

[0056] Clustering the plurality of image regions to obtain a plurality of first clustering results, and clustering the plurality of third images to obtain a plurality of second clustering results;

[0057] A plurality of image regions are selected from the plurality of first clustering results as the plurality of foreground images, and a plurality of third images are selected from the plurality of second clustering results as the plurality of background images.

[0058] Optionally, the fusion module is specifically used for:

[0059] Determine the acquisition time of a frame of image where the multiple foreground images are located in the video stream, and the positions of the multiple foreground images in the frame of image where the multiple foreground images are located, so as to obtain the acquisition time and image positions corresponding to the multiple foreground images respectively;

[0060] Performing semantic segmentation on the multiple background images respectively to determine semantic segmentation information corresponding to the multiple background images respectively;

[0061] Determine the acquisition time of a frame of image where the multiple background images are located in the video stream to obtain the acquisition time corresponding to the multiple background images respectively;

[0062] The multiple foreground images and the multiple background images are fused based on the acquisition time and image positions respectively corresponding to the multiple foreground images and the acquisition time and semantic segmentation information respectively corresponding to the multiple background images.

[0063] On the other hand, a computer device is provided, which is a camera or a server, and includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of the above-mentioned method for determining a sample image.

[0064] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the method for determining a sample image are implemented.

[0065] In another aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer, the computer is caused to execute the steps of the method for determining a sample image.

[0066] The technical solution provided by the embodiments of the present application can at least bring the following beneficial effects:

[0067] The embodiment of the present application performs false alarm and omission analysis on the video stream to determine the first image and the false alarm and omission results corresponding to the first image. In this way, multiple foreground images and multiple background images are obtained from the video stream through the false alarm and omission results corresponding to the first image. The multiple foreground images and the multiple background images are then fused to obtain multiple second images. Finally, the first image and the multiple second images are determined as sample images. In this way, the embodiment of the present application generates more sample images, and the first neural network model is trained by these sample images, which can improve the training effect of the first neural network model, thereby effectively improving the analysis performance of the first neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0070] Figure 2 is a structural schematic diagram of a sample image determination device provided in an embodiment of the present application;

[0071] Figure 3 is a flow chart of a method for determining a sample image provided by an embodiment of the present application;

[0072] Figure 4 is a structural schematic diagram of a device for determining a sample image provided by an embodiment of the present application;

[0073] Figure 5 It is a structural schematic diagram of a camera provided in an embodiment of the present application;

[0074] Figure 6 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0076] Before explaining in detail the method for determining a sample image provided in the embodiment of the present application, the application scenario and implementation environment provided in the embodiment of the present application are first introduced.

[0077] In order to realize the analysis of the target object, it is usually necessary to train the first neural network model through sample images. Afterwards, the target object in the video stream captured by the camera can be analyzed by the trained first neural network model. However, in the process of the first neural network model analyzing the target object, it will continue to be updated, and with the update of the first neural network model and the increase in the types of target objects learned by the first neural network model, the first neural network model may forget certain types of target objects that have been learned before. When it is necessary to analyze these types of target objects again, there may be a problem of low analysis accuracy. In layman's terms, due to the continuous updating of the first neural network model, the characteristics of certain types of target objects may be forgotten, and these target objects cannot be analyzed, or the analysis accuracy of these target objects is relatively low.

[0078] For example, the first neural network model needs to recognize the characteristics of the target object in the four seasons. However, after the four seasons of spring, summer, autumn and winter, if the target object needs to be analyzed again in spring, since the first neural network model is updated as the characteristics of the target object in the four seasons change, the first neural network model may forget the characteristics of the target object in spring. Therefore, when analyzing the target object in spring, it may cause the problem of low analysis accuracy.

[0079] Therefore, in the process of analyzing the target object through the first neural network model, the sample image can also be determined and stored according to the method provided in the embodiment of the present application based on the video stream captured by the camera, that is, while the target object is analyzed online through the first neural network model, the sample image can also be determined and stored. In this way, each time the target object is analyzed subsequently, the first neural network model can be retrained based on the sample image, thereby improving the analysis accuracy of the first neural network model.

[0080] Please refer to Figure 1 , Figure 1 is a schematic diagram of an implementation environment according to an exemplary embodiment. The implementation environment includes at least one camera 101 ( Figure 1 In the figure, a camera is used to schematically represent the at least one camera) and the server 102, and the camera 101 can be communicatively connected with the server 102. The communication connection can be a wired or wireless connection, which is not limited in the embodiment of the present application.

[0081] The camera 101 is used to capture a video stream and transmit the captured video stream to the server 102 .

[0082] The server 102 is deployed with a first neural network model, through which the target object in the video stream transmitted by the camera 101 can be analyzed. At the same time, the server 102 can also determine and store sample images based on the video stream transmitted by the camera 101.

[0083] Of course, the above is an exemplary implementation environment, and in other implementation environments, the server 102 may not be included. In this way, the camera 101 can shoot a video stream and analyze the target object in the video stream according to the method provided in the embodiment of the present application, and then determine and store the sample image.

[0084] That is, in the embodiment of the present application, the camera 101 may process the video stream to determine the sample image, or the server 102 may process the video stream to determine the sample image. For ease of description, the camera or the server may be referred to as a sample image determination device.

[0085] For example, please refer to Figure 2 , the sample image determination device may include a false alarm analysis module, a spatiotemporal mining module, a storage module and a sample generation module. The false alarm analysis module is used to determine an image with a false alarm object as a first image from a video stream during the analysis of a target object, and simultaneously determine a false alarm result corresponding to the first image. The spatiotemporal mining module is used to obtain a plurality of foreground images and a plurality of background images from a video stream based on the false alarm result corresponding to the first image. The storage module is used to store the first image, and the plurality of foreground images and the plurality of background images obtained by mining. The sample generation module is used to fuse the plurality of foreground images and the plurality of background images stored to obtain a plurality of second images, and then determine the first image and the plurality of second images as sample images.

[0086] It should be noted that the storage module may not store multiple foreground images and multiple background images, but after mining to obtain multiple foreground images and multiple background images, the multiple foreground images and the multiple background images are first merged through the sample generation module to obtain multiple second images, and then the multiple second images are stored in the storage module.

[0087] Of course, the storage module can also store multiple foreground images and multiple background images, and store multiple second images at the same time. In this way, other images can be generated based on the multiple foreground images and multiple background images for training the first neural network model.

[0088] The camera 101 may be any device with a camera function, such as a smart phone, a digital camera, a PTZ monitoring device, etc. The server 102 may be a single server, or a server cluster consisting of multiple servers, or a cloud computing service center.

[0089] Those skilled in the art should understand that the above-mentioned camera 101 and server 102 are only examples, and other existing or future cameras or servers that are applicable to the embodiments of the present application should also be included in the protection scope of the embodiments of the present application and are included here by reference.

[0090] Next, the method for determining the sample image provided in the embodiment of the present application is explained in detail.

[0091] Figure 3 This is a flow chart of a method for determining a sample image provided by an embodiment of the present application, and the method is applied to a server. Figure 3 , the method comprises the following steps.

[0092] Step 301: Perform false alarm analysis on the video stream to determine a first image and a false alarm result corresponding to the first image, where the first image is an image in the video stream where a false alarm object exists, and the false alarm object is a target object that is missed in analysis or a target object that is erroneously analyzed.

[0093] In some embodiments, a plurality of first neural network models and a second neural network model can be used to perform false alarm analysis on any frame image in the video stream to determine whether there is a false alarm object in the frame image, and the second neural network model is a model that can analyze all target objects in the frame image. If there is a false alarm object in the frame image, the frame image is used as the first image, and the false alarm result corresponding to the first image is determined.

[0094] It should be noted that the above video stream may be a video stream captured by any one of the at least one camera.

[0095] Since a video stream may contain multiple frames of images, for any frame of image, it is possible to determine whether there is a missed false alarm object in the image according to the above method, and then determine whether to use the image as the first image. After each frame of image in the video stream is processed according to the above method, multiple first images and missed false alarm results corresponding to each first image can be obtained. The embodiment of the present application is described by taking a first image as an example.

[0096] Among them, the false alarm result may include the location information of at least one false alarm object in the first image, and the acquisition time of the first image. Of course, the false alarm result may also include other information, which is not limited in the embodiments of the present application. In addition, based on the above description, the false alarm object may be a target object that is missed for analysis, or a target object that is erroneously analyzed, so for the at least one false alarm object, the at least one false alarm object may include a target object that is missed for analysis, or may include a target object that is erroneously analyzed. Moreover, in the case where the first image includes multiple false alarm objects, the multiple false alarm objects may include both target objects that are missed for analysis and target objects that are erroneously analyzed.

[0097] It should be noted that the multiple first neural network models can all analyze the target object, and the multiple first neural network models can be network models with different structures, and of course, they can also be network models with the same structure. Moreover, each first neural network model refers to a model that has been trained with a data set. When the multiple first neural network models have the same structure, the multiple first neural network models can be trained with different data sets. When the multiple first neural network models have different structures, the multiple first neural network models can be trained with the same data set or with different data sets. The second neural network model is trained with a variety of public data sets. The second neural network model is a model that can analyze all target objects in any frame image, and the second neural network model can basically recognize all target objects. For example, the second neural network model is an open set recognition model.

[0098] Among them, the implementation process of performing false alarm and omission analysis on any frame image in the video stream through multiple first neural network models and second neural network models to determine whether there is a false alarm object in any frame image includes: determining the first analysis result corresponding to any frame image through multiple first neural network models respectively to obtain multiple first analysis results, the first analysis result includes the position information of at least one target object in the any frame image and the first label of each target object. Determine the second analysis result corresponding to any frame image through the second neural network model, the second analysis result includes the position information of at least one target object in the any frame image and the second label of each target object. Based on the position information and first label of the target object included in the multiple first analysis results, and the position information and second label of the target object included in the second analysis result, determine whether there is a false alarm object in any frame image.

[0099] Since each of the multiple first neural network models determines the first analysis result corresponding to any frame image in the same way, one of the first neural network models is used as an example for explanation. That is, the any frame image is used as the input of the first neural network model to obtain the position information of at least one target object output by the first neural network model, and the probability that each target object belongs to multiple labels. For each target object in the at least one target object, the maximum probability is determined from the probabilities that the target object belongs to multiple labels, and the label corresponding to the maximum probability is used as the first label corresponding to the target object. At this time, the position information of the at least one target object in the any frame image and the first label corresponding to each target object can be obtained, that is, the first analysis result corresponding to the any frame image. After the any frame image is processed by the multiple first neural network models respectively, multiple first analysis results can be obtained.

[0100] Similarly, the implementation process of determining the second analysis result corresponding to any frame image through the second neural network model includes: using the any frame image as the input of the second neural network model to obtain the position information of at least one target object output by the second neural network model, and the probability that each target object belongs to multiple labels. For each target object in the at least one target object, the maximum probability is determined from the probabilities that the target object belongs to multiple labels, and the label corresponding to the maximum probability is used as the second label corresponding to the target object. At this time, the position information of the at least one target object in the any frame image and the second label corresponding to each target object can be obtained, that is, the second analysis result corresponding to the any frame image.

[0101] Among them, based on the position information and the first label of the target object included in the multiple first analysis results, and the position information and the second label of the target object included in the second analysis result, the implementation process of determining whether there is a missed false alarm object in any frame image can be: if the position information and the first label of the target object included in the multiple first analysis results are the same, and are the same as the position information and the second label of the target object included in the second analysis result, then it is determined that there is no missed false alarm object in any frame image; otherwise, it is determined that there is a missed false alarm object in any frame image.

[0102] Among them, after determining multiple first analysis results corresponding to any frame image through multiple first neural network models, for any two first analysis results among the multiple first analysis results, at least one target object included in the two first analysis results may be the same or different, and the first label corresponding to the same target object may be the same or different. Similarly, after determining the second analysis result corresponding to any frame image through the second neural network model, at least one target object included in the second analysis result may be the same or different from at least one target object included in any of the above-mentioned first analysis results, and the first label corresponding to the same target object may be the same or different.

[0103] Moreover, based on the above description, the second neural network model can identify all target objects in any one of the images, so the multiple first analysis results and the second analysis results can be compared to determine whether any one of the frame images includes a missed or falsely reported object. As an example, for any one of the target objects included in the multiple first analysis results, determine whether the multiple first analysis results all include the location information and the first label of the target object. If the multiple first analysis results all include the location information and the first label of the target object, determine that the target object is not a target object that was missed in the analysis; otherwise, determine that the target object is a target object that was missed in the analysis.

[0104] In the case where the target object is not a target object that is missed in the analysis, it can also be determined whether the location information and the first label of the target object included in the multiple first analysis results are the same, and whether the location information and the first label of the target object are the same as the location information and the second label of the target object in the second analysis result. If the location information and the first label of the target object included in the multiple first analysis results are the same, and the location information and the first label of the target object are the same as the location information and the second label of the target object in the second analysis result, it is determined that the target object is not a target object that is misanalyzed, otherwise, it is determined that the target object is a target object that is misanalyzed.

[0105] Optionally, there may be many images in the video stream that have missed or falsely reported objects. In this case, the score of each of these images may be determined based on the missed or falsely reported objects in these images. Afterwards, some images are selected as the first images according to the scores.

[0106] For example, the scores of each image can be determined based on the number and importance of missed or falsely reported objects, and then the image with a score higher than the score threshold is selected as the first image. Of course, the scores of each image can also be determined in other ways, and the first image can also be selected in other ways.

[0107] Step 302: Based on the false alarm and omission results corresponding to the first image, a plurality of foreground images and a plurality of background images are obtained from the video stream.

[0108] In some embodiments, the false alarm result includes the location information of at least one false alarm object in the first image and the acquisition time of the first image. In this way, at least one spatial range can be determined based on the location information of the at least one false alarm object, and the at least one spatial range corresponds to the at least one false alarm object. A time range is determined based on the acquisition time of the first image. Based on the time range and the at least one spatial range, multiple foreground images and multiple background images are obtained from the video stream.

[0109] As an example, a first time and a second time may be determined, wherein the first time is located before the acquisition time of the first image and is separated from the acquisition time of the first image by a first time length, and the second time is located after the acquisition time of the first image and is separated from the acquisition time of the first image by a second time length. The time range between the first time and the second time is determined as the time range corresponding to the first image.

[0110] The first duration and the second duration can be set in advance and adjusted according to different requirements. The first duration and the second duration can be equal or unequal.

[0111] For any missed false alarm object in the first image, the geometric center of the missed false alarm object can be determined based on the position information of the missed false alarm object. Then, a circular area with the geometric center of the missed false alarm object as the center and including the missed false alarm object is determined from the first image, and the circular area is used as a spatial range. Of course, a region including the at least one missed false alarm object can also be determined from the first image, and the region is used as the spatial range corresponding to the first image.

[0112] As an example, the geometric centers of the missed false alarm objects can be determined based on the positions of the missed false alarm objects in the first image. Then, a circular area with the geometric center as the center and including the missed false alarm objects is determined from the first image, and the circular area is used as the spatial range corresponding to the first image.

[0113] Based on the time range and the at least one spatial range, the implementation process of obtaining multiple foreground images and multiple background images from the video stream includes: obtaining an image area whose acquisition time is within the time range, whose position is within the at least one spatial range and which is the foreground from the video stream to obtain multiple image areas. Obtaining an image whose acquisition time is within the time range and which does not exist in the foreground from the video stream to obtain multiple third images. Clustering the multiple image areas to obtain multiple first clustering results, and clustering the multiple third images to obtain multiple second clustering results. Selecting multiple image areas from the multiple first clustering results as multiple foreground images, and selecting multiple third images from the multiple second clustering results as multiple background images.

[0114] That is, image regions whose acquisition time is within the time range and whose position is within any spatial range of the at least one spatial range and which are foregrounds are obtained, and then these image regions are clustered to determine a plurality of foreground images. Also, images whose acquisition time is within the time range and whose position is within any spatial range of the at least one spatial range and which do not have foregrounds are obtained to obtain a plurality of third images, and then these third images are clustered to determine a plurality of background images.

[0115] It should be noted that clustering refers to the process of classification based on the similarity of image features, and after clustering, the similarity of image features in the same clustering result is as large as possible, and the difference of image features in different clustering results is as large as possible. Therefore, after clustering multiple image regions, the same first clustering result includes image regions with similar image features, and different first clustering results include image regions with dissimilar image features. Similarly, after clustering multiple third images, the same second clustering result includes third images with similar image features, and different second clustering results include third images with dissimilar image features.

[0116] Since the first neural network model processes any frame of the video stream, the first neural network model will extract image features of each image when performing target object analysis. In this way, when clustering, the image features can be features output by the first neural network model or re-extracted features, which is not limited in the present embodiment of the application.

[0117] After clustering the above-mentioned multiple image regions, the same first clustering result includes image regions with similar image features, and different first clustering results include image regions with dissimilar image features. Therefore, in order to ensure the richness and diversity of sample images, one or more image regions can be selected from each first clustering result as multiple foreground images. Similarly, after clustering the above-mentioned multiple third images, the same second clustering result includes third images with similar image features, and different second clustering results include third images with dissimilar image features. Therefore, in order to ensure the richness and diversity of sample images, one or more third images can be selected from each second clustering result as multiple background images.

[0118] In the case where there are missed false positive objects in the first image, it indicates that the multiple first neural network models have forgotten the features of the target object in the first image. In a video stream, the contents of multiple consecutive frames of images may be similar. At this time, there may also be features in other images that are forgotten by multiple first neural network models. Therefore, a time range can be determined based on the acquisition time of the first image according to the above method, and at least one spatial range can be determined based on the position of at least one missed false positive object. In this way, the foreground image where multiple target objects with features that may be forgotten are located can be determined through the time range and the at least one spatial range. At the same time, in order to facilitate subsequent image fusion, multiple background images can also be determined based on the time range and the at least one spatial range, so that more sample images can be obtained after subsequent image fusion, which fully guarantees the richness of the sample images and makes the determined sample images more valuable.

[0119] Step 303: Fusing the multiple foreground images and the multiple background images to obtain multiple second images.

[0120] In some embodiments, the acquisition time of a frame image in which the multiple foreground images are located in the video stream, and the positions of the multiple foreground images in the frame image in which they are located can be determined to obtain the acquisition time and image position corresponding to the multiple foreground images respectively. The multiple background images are semantically segmented to determine the semantic segmentation information corresponding to the multiple background images respectively. The acquisition time of a frame image in which the multiple background images are located in the video stream is determined to obtain the acquisition time corresponding to the multiple background images respectively. Based on the acquisition time and image position corresponding to the multiple foreground images respectively, and the acquisition time and semantic segmentation information corresponding to the multiple background images respectively, the multiple foreground images and the multiple background images are fused.

[0121] Since a foreground image is a partial image area in a frame of an image in a video stream, after the foreground image is determined, the acquisition time of the frame of an image in which the foreground image is located in the video stream and the position of the foreground image in the frame of an image can be determined, and then the acquisition time is determined as the acquisition time corresponding to the foreground image, and the position of the foreground image in the frame of an image is determined as the image position corresponding to the foreground image. Similarly, a background image is a frame of an image in a video stream, and therefore, after the background image is determined, the acquisition time of the frame of an image in which the background image is located in the video stream can be determined, and then the acquisition time is determined as the acquisition time corresponding to the background image.

[0122] It should be noted that, for any background image among the multiple background images, the background image includes multiple pixel points. The semantic segmentation algorithm can be used to segment each pixel point in the background image according to different semantic expressions, thereby obtaining the semantic segmentation information of the background image. The semantic segmentation information includes semantic information of different areas in the background image, such as semantic information of the sky, grass, etc.

[0123] Based on the acquisition time and image position corresponding to the multiple foreground images, and the acquisition time and semantic segmentation information corresponding to the multiple background images, the implementation process of fusing the multiple foreground images with the multiple background images includes: for any foreground image among the multiple foreground images, selecting at least one background image whose acquisition time is within the same time period as the acquisition time of the foreground image from the multiple background images. Based on the image position corresponding to the foreground image and the semantic segmentation information of the at least one background image, determining a target background image from the at least one background image, and fusing the foreground image with the target background image so that the position of the foreground image in the target background image is the image position corresponding to the foreground image.

[0124] The same time period refers to the same period of the same season, such as daytime in spring, nighttime in spring, daytime in winter, nighttime in winter, and so on.

[0125] Since the background image may include semantic segmentation areas such as the sky, grass, and rivers, and the foreground image may not be suitable to be located in certain semantic segmentation areas, for example, people are not suitable to be located in the sky. Therefore, the semantic segmentation information of the semantic segmentation areas in which the objects included in the foreground image are allowed to appear can be determined to obtain at least one target semantic segmentation information. Then, from the at least one background image, the semantic segmentation information of the semantic segmentation area located at the image position corresponding to the foreground image is selected as the background image of any target semantic segmentation information, and the selected background image is used as the target background image.

[0126] Among them, the semantic segmentation information of the semantic segmentation areas where objects of different categories are allowed to appear can be stored in advance, so that the category of the object included in the foreground image can be determined, and then the semantic segmentation information of the semantic segmentation areas where the object is allowed to appear can be determined.

[0127] Optionally, when fusing the foreground image with the target background image, the parts of the object in the foreground image may be segmented, and based on the segmented parts, the foreground image and the target background image may be fused so that some parts of the object may be hidden behind an object in the target background image, thereby generating a more valuable image. For example, a person is selected as a foreground image and a fence is selected as a background image. When fusing the person and the fence, Gaussian transparency may be added to the leg and foot regions of the person in the foreground image, and then the foreground image is fused with the fence. The fused image shows that the leg and foot regions of the person are hidden behind the fence.

[0128] Step 304: Determine the first image and the plurality of second images as sample images.

[0129] It should be noted that the above is an example of a video stream shot by a camera to introduce the implementation process of determining the sample image. In practical applications, the video streams shot by multiple cameras can also be processed to determine the sample image. The implementation process of processing the video streams shot by multiple cameras is similar to the implementation process of processing the video streams mentioned above, but it is worth noting that when generating the second image, the foreground image and the background image in the video stream shot by the same camera need to be fused.

[0130] In an embodiment of the present application, in the process of analyzing the target object in the video stream, the first image can also be determined. Since there are missed false alarm objects in the first image, that is, the analysis result of the target object of the first image by the first neural network model is inaccurate, so it is more targeted and more valuable to train the first neural network model again through the first image, thereby improving the training efficiency and training effect. In addition, on the basis of generating the first image, the embodiment of the present application can also obtain more valuable foreground images and background images, and then fuse these foreground images and background images to generate the second image, thereby increasing the number of sample images generated. Moreover, by clustering multiple image regions to select different types of foreground images, and clustering multiple third images to select different types of background images, the types of the generated second images can be made richer, which can further improve the training effect of the first neural network model, thereby effectively improving the analysis performance of the first neural network model.

[0131] Figure 4is a schematic diagram of a structure of a device for determining a sample image provided by an embodiment of the present application. The device for determining a sample image can be implemented by software, hardware, or a combination of both to form part or all of a computer device. The computer device can be Figure 1 Please refer to the camera or server shown. Figure 4 The device includes: a first determination module 401, an acquisition module 402, a fusion module 403 and a second determination module 404.

[0132] A first determination module 401 is used to perform a missed false alarm analysis on the video stream to determine a first image and a missed false alarm result corresponding to the first image, wherein the first image is an image in the video stream where a missed false alarm object exists, and the missed false alarm object is a target object that is missed in the analysis or a target object that is wrongly analyzed;

[0133] An acquisition module 402, configured to acquire a plurality of foreground images and a plurality of background images from a video stream based on the false alarm result corresponding to the first image;

[0134] A fusion module 403, used for fusing the multiple foreground images and the multiple background images to obtain multiple second images;

[0135] The second determining module 404 is configured to determine the first image and a plurality of second images as sample images.

[0136] Optionally, the first determining module 401 includes:

[0137] A first determination submodule is used to perform a false alarm analysis on any frame image in the video stream through a plurality of first neural network models and a second neural network model to determine whether there is a false alarm object in the any frame image, and the second neural network model is a model capable of analyzing all target objects in the any frame image;

[0138] The second determination submodule is used to, if there is a missed or false alarm object in any frame of image, take the any frame of image as the first image and determine the missed or false alarm result corresponding to the first image.

[0139] Optionally, the first determining submodule includes:

[0140] A first determination unit is used to determine the first analysis results corresponding to any frame image respectively through multiple first neural network models to obtain multiple first analysis results, where the first analysis results include position information of at least one target object in the any frame image and a first label of each target object;

[0141] A second determination unit is used to determine a second analysis result corresponding to the any frame image through a second neural network model, where the second analysis result includes position information of at least one target object in the any frame image and a second label of each target object;

[0142] The third determination unit is used to determine whether there is a missed or falsely reported object in any frame image based on the position information and the first label of the target object included in the multiple first analysis results, and the position information and the second label of the target object included in the second analysis result.

[0143] Optionally, the third determining unit is specifically configured to:

[0144] If the location information and the first label of the target object included in multiple first analysis results are the same, and are the same as the location information and the second label of the target object included in the second analysis result, it is determined that there is no missed false alarm object in any frame image; otherwise, it is determined that there is a missed false alarm object in any frame image.

[0145] Optionally, the missed false alarm result includes location information of at least one missed false alarm object in the first image, and acquisition time of the first image;

[0146] The acquisition module 402 includes:

[0147] A third determination submodule is used to determine at least one spatial range based on the location information of the at least one missed or falsely reported object, and the at least one spatial range corresponds to the at least one missed or falsely reported object in a one-to-one manner;

[0148] A fourth determination submodule, configured to determine a time range based on the acquisition time of the first image;

[0149] The acquisition submodule is used to acquire multiple foreground images and multiple background images from the video stream based on the time range and the at least one spatial range.

[0150] Optionally, obtain the submodule specifically for:

[0151] Acquire from the video stream an image region whose acquisition time is within the time range, whose position is within the at least one spatial range and which is a foreground, to obtain a plurality of image regions;

[0152] Acquire images whose acquisition time is within the time range and without foreground from the video stream to obtain a plurality of third images;

[0153] Clustering the plurality of image regions to obtain a plurality of first clustering results, and clustering the plurality of third images to obtain a plurality of second clustering results;

[0154] A plurality of image regions are selected from the plurality of first clustering results as a plurality of foreground images, and a plurality of third images are selected from the plurality of second clustering results as a plurality of background images.

[0155] Optionally, the fusion module 403 is specifically used for:

[0156] Determine the acquisition time of a frame of image where the multiple foreground images are located in the video stream, and the positions of the multiple foreground images in the frame of image where the multiple foreground images are located, so as to obtain the acquisition time and image positions corresponding to the multiple foreground images respectively;

[0157] Performing semantic segmentation on the multiple background images respectively to determine semantic segmentation information corresponding to the multiple background images respectively;

[0158] Determine the acquisition time of a frame of image where multiple background images are located in the video stream to obtain the acquisition time corresponding to the multiple background images respectively;

[0159] Based on the acquisition time and image positions respectively corresponding to the multiple foreground images and the acquisition time and semantic segmentation information respectively corresponding to the multiple background images, the multiple foreground images and the multiple background images are fused.

[0160] In an embodiment of the present application, in the process of analyzing the target object in the video stream, the first image can also be determined. Since there are missed false alarm objects in the first image, that is, the analysis result of the target object of the first image by the first neural network model is inaccurate, so it is more targeted and more valuable to train the first neural network model again through the first image, thereby improving the training efficiency and training effect. In addition, on the basis of generating the first image, the embodiment of the present application can also obtain more valuable foreground images and background images, and then fuse these foreground images and background images to generate the second image, thereby increasing the number of sample images generated. Moreover, by clustering multiple image regions to select different types of foreground images, and clustering multiple third images to select different types of background images, the types of the generated second images can be made richer, which can further improve the training effect of the first neural network model, thereby effectively improving the analysis performance of the first neural network model.

[0161] It should be noted that: the device for determining a sample image provided in the above embodiment only uses the division of the above functional modules as an example when determining a sample image. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining a sample image provided in the above embodiment and the method embodiment for determining a sample image belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0162] Figure 51 is a block diagram of a terminal 500 provided in an embodiment of the present application. In an embodiment of the present application, the terminal 500 can be used as a camera. The terminal 500 can be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 500 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0163] Typically, the terminal 500 includes a processor 501 and a memory 502 .

[0164] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0165] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement the method for determining a sample image provided in the method embodiment of the present application.

[0166] In some embodiments, the terminal 500 may further optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504, a touch display screen 505, a camera 506, an audio circuit 507, a positioning component 508 and a power supply 509.

[0167] The peripheral device interface 503 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502. In some embodiments, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0168] The radio frequency circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 504 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 504 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 504 may also include circuits related to NFC (Near Field Communication), which is not limited in the embodiments of the present application.

[0169] The display screen 505 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to collect touch signals on the surface or above the surface of the display screen 505. The touch signal can be input to the processor 501 as a control signal for processing. At this time, the display screen 505 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 505 can be one, and the front panel of the terminal 500 is set; in other embodiments, the display screen 505 can be at least two, which are respectively set on different surfaces of the terminal 500 or are folded; in some other embodiments, the display screen 505 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 500. Even, the display screen 505 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 505 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0170] The camera assembly 506 is used to capture images or videos. Optionally, the camera assembly 506 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0171] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 501 for processing, or input them into the radio frequency circuit 504 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 500. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 507 may also include a headphone jack.

[0172] The positioning component 508 is used to locate the current geographical location of the terminal 500 to implement navigation or LBS (Location Based Service). The positioning component 508 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or Russia's Galileo system.

[0173] The power supply 509 is used to power various components in the terminal 500. The power supply 509 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 509 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0174] In some embodiments, the terminal 500 further includes one or more sensors 510 , including but not limited to: an acceleration sensor 511 , a gyroscope sensor 512 , a pressure sensor 513 , a fingerprint sensor 514 , an optical sensor 515 , and a proximity sensor 516 .

[0175] The acceleration sensor 511 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 500. For example, the acceleration sensor 511 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 501 can control the touch screen 505 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 511. The acceleration sensor 511 can also be used for collecting game or user motion data.

[0176] The gyro sensor 512 can detect the body direction and rotation angle of the terminal 500, and the gyro sensor 512 can cooperate with the acceleration sensor 511 to collect the user's 3D actions on the terminal 500. The processor 501 can implement the following functions based on the data collected by the gyro sensor 512: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0177] The pressure sensor 513 can be set on the side frame of the terminal 500 and / or the lower layer of the touch display screen 505. When the pressure sensor 513 is set on the side frame of the terminal 500, it can detect the user's holding signal of the terminal 500, and the processor 501 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 513. When the pressure sensor 513 is set on the lower layer of the touch display screen 505, the processor 501 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 505. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0178] The fingerprint sensor 514 is used to collect the user's fingerprint, and the processor 501 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 501 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 514 can be set on the front, back, or side of the terminal 500. When a physical button or a manufacturer logo is set on the terminal 500, the fingerprint sensor 514 can be integrated with the physical button or the manufacturer logo.

[0179] The optical sensor 515 is used to collect the ambient light intensity. In one embodiment, the processor 501 can control the display brightness of the touch display screen 505 according to the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 505 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 505 is reduced. In another embodiment, the processor 501 can also dynamically adjust the shooting parameters of the camera assembly 506 according to the ambient light intensity collected by the optical sensor 515.

[0180] The proximity sensor 516, also called a distance sensor, is usually arranged on the front panel of the terminal 500. The proximity sensor 516 is used to collect the distance between the user and the front of the terminal 500. In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the terminal 500 is gradually decreasing, the processor 501 controls the touch display screen 505 to switch from the screen-on state to the screen-off state; when the proximity sensor 516 detects that the distance between the user and the front of the terminal 500 is gradually increasing, the processor 501 controls the touch display screen 505 to switch from the screen-off state to the screen-on state.

[0181] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the terminal 500, and the terminal 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0182] Figure 6 600 includes a central processing unit (CPU) 601, a system memory 604 including a random access memory (RAM) 602 and a read-only memory (ROM) 603, and a system bus 605 connecting the system memory 604 and the central processing unit 601. The server 600 also includes a basic input / output system (I / O system) 606 that helps transmit information between various devices in the computer, and a large-capacity storage device 607 for storing an operating system 613, an application program 614, and other program modules 615.

[0183] The basic input / output system 606 includes a display 608 for displaying information and an input device 609 such as a mouse and a keyboard for user inputting information. The display 608 and the input device 609 are connected to the central processing unit 601 through an input / output controller 610 connected to the system bus 605. The basic input / output system 606 may also include an input / output controller 610 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 610 also provides output to a display screen, a printer, or other types of output devices.

[0184] The mass storage device 607 is connected to the central processing unit 601 through a mass storage controller (not shown) connected to the system bus 605. The mass storage device 607 and its associated computer-readable media provide non-volatile storage for the server 600. That is, the mass storage device 607 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.

[0185] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 604 and mass storage device 607 can be collectively referred to as memory.

[0186] According to various embodiments of the present application, the server 600 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 600 can be connected to the network 612 through the network interface unit 611 connected to the system bus 605, or the network interface unit 611 can be used to connect to other types of networks or remote computer systems (not shown).

[0187] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0188] In some embodiments, a computer-readable storage medium is further provided, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for determining a sample image in the above embodiment are implemented. For example, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0189] It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0190] It should be understood that all or part of the steps to implement the above embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-mentioned computer readable storage medium.

[0191] That is, in some embodiments, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the method for determining a sample image as described above.

[0192] It should be understood that the "at least one" mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0193] The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining a sample image, characterized in that: The method comprises: Performing omission and false alarm analysis on the video stream to determine a first image and an omission and false alarm result corresponding to the first image, wherein the first image is an image in the video stream where an omission and false alarm object exists, the omission and false alarm object is a target object that is omission-analyzed or a target object that is erroneously analyzed, and the omission and false alarm result includes location information of at least one omission and false alarm object in the first image and acquisition time of the first image; Determine at least one spatial range based on the location information of the at least one missed or falsely reported object, the at least one spatial range corresponding to the at least one missed or falsely reported object one-to-one; determining a time range based on the acquisition time of the first image; Based on the time range and the at least one spatial range, acquiring a plurality of foreground images and a plurality of background images from the video stream; fusing the plurality of foreground images and the plurality of background images to obtain a plurality of second images; The first image and the plurality of second images are determined as sample images.

2. The method according to claim 1, characterized in that The performing false alarm and omission analysis on the video stream to determine the first image and the false alarm and omission result corresponding to the first image includes: By using a plurality of first neural network models and a second neural network model, a false alarm and omission analysis is performed on any frame image in the video stream to determine whether there is a false alarm and omission object in any frame image, wherein the second neural network model is a model capable of analyzing all target objects in any frame image; If there is a false alarm object in any of the frame images, the any frame image is used as the first image, and the false alarm result corresponding to the first image is determined.

3. The method according to claim 2, characterized in that The method of performing a false alarm analysis on any frame image in the video stream by using a plurality of first neural network models and a second neural network model to determine whether there is a false alarm object in any frame image includes: Determine the first analysis results corresponding to any one of the frames of image respectively through the multiple first neural network models to obtain multiple first analysis results, wherein the first analysis results include position information of at least one target object in any one of the frames of image and a first label of each target object; Determine a second analysis result corresponding to any one of the frame images by using the second neural network model, wherein the second analysis result includes location information of at least one target object in any one of the frame images and a second label of each target object; Based on the location information and the first label of the target object included in the plurality of first analysis results, and the location information and the second label of the target object included in the second analysis result, it is determined whether there is a missed or falsely reported object in any frame of image.

4. The method according to claim 3, characterized in that The determining whether there is a missed or falsely reported object in any frame image based on the location information and the first label of the target object included in the multiple first analysis results, and the location information and the second label of the target object included in the second analysis result, comprises: If the location information and the first label of the target object included in the multiple first analysis results are the same, and are the same as the location information and the second label of the target object included in the second analysis result, it is determined that there is no missed false alarm object in any of the frame images; otherwise, it is determined that there is a missed false alarm object in any of the frame images.

5. The method according to claim 1, characterized in that The acquiring a plurality of foreground images and a plurality of background images from the video stream based on the time range and the at least one spatial range comprises: Acquire from the video stream an image region whose acquisition time is within the time range, whose position is within the at least one space range and which is a foreground, so as to obtain a plurality of image regions; Acquire from the video stream an image whose acquisition time is within the time range and does not have a foreground, so as to obtain a plurality of third images; Clustering the plurality of image regions to obtain a plurality of first clustering results, and clustering the plurality of third images to obtain a plurality of second clustering results; A plurality of image regions are selected from the plurality of first clustering results as the plurality of foreground images, and a plurality of third images are selected from the plurality of second clustering results as the plurality of background images.

6. The method according to claim 1, characterized in that The fusing the multiple foreground images and the multiple background images comprises: Determine the acquisition time of a frame of image where the multiple foreground images are located in the video stream, and the positions of the multiple foreground images in the frame of image where the multiple foreground images are located, so as to obtain the acquisition time and image positions corresponding to the multiple foreground images respectively; Performing semantic segmentation on the multiple background images respectively to determine semantic segmentation information corresponding to the multiple background images respectively; Determine the acquisition time of a frame of image where the multiple background images are located in the video stream to obtain the acquisition time corresponding to the multiple background images respectively; The multiple foreground images and the multiple background images are fused based on the acquisition time and image positions respectively corresponding to the multiple foreground images and the acquisition time and semantic segmentation information respectively corresponding to the multiple background images.

7. A device for determining a sample image, characterized in that: The device comprises: A first determination module is used to perform a missed false alarm analysis on the video stream to determine a first image and a missed false alarm result corresponding to the first image, the first image is an image in the video stream where a missed false alarm object exists, the missed false alarm object is a target object that is missed in analysis or a target object that is erroneously analyzed, and the missed false alarm result includes location information of at least one missed false alarm object in the first image and a capture time of the first image; The acquisition module includes a third determination submodule, a fourth determination submodule and an acquisition submodule, wherein the third determination submodule is used to determine at least one spatial range based on the position information of the at least one missed false alarm object, and the at least one spatial range corresponds to the at least one missed false alarm object; the fourth determination submodule is used to determine a time range based on the acquisition time of the first image; and the acquisition submodule is used to acquire a plurality of foreground images and a plurality of background images from the video stream based on the time range and the at least one spatial range; A fusion module, used for fusing the multiple foreground images and the multiple background images to obtain multiple second images; The second determining module is configured to determine the first image and the plurality of second images as sample images.

8. The device according to claim 7, characterized in that The first determining module comprises: A first determination submodule is used to perform a false alarm analysis on any frame image in the video stream through a plurality of first neural network models and a second neural network model to determine whether there is a false alarm object in any frame image, wherein the second neural network model is a model capable of analyzing all target objects in any frame image; The second determination submodule is used to, if there is a missed or false alarm object in any frame of image, use the any frame of image as the first image and determine the missed or false alarm result corresponding to the first image.

9. The device according to claim 8, characterized in that The first determining submodule includes: A first determination unit is used to determine the first analysis results corresponding to any one of the frame images respectively through the multiple first neural network models to obtain multiple first analysis results, wherein the first analysis results include position information of at least one target object in any one of the frame images and a first label of each target object; A second determining unit, configured to determine a second analysis result corresponding to any one frame of image by using the second neural network model, wherein the second analysis result includes position information of at least one target object in any one frame of image and a second label of each target object; The third determination unit is used to determine whether there is a missed or falsely reported object in any frame image based on the position information and the first label of the target object included in the multiple first analysis results, and the position information and the second label of the target object included in the second analysis result.

10. The device according to claim 9, characterized in that The third determining unit is specifically configured to: If the location information and the first label of the target object included in the multiple first analysis results are the same, and are the same as the location information and the second label of the target object included in the second analysis result, it is determined that there is no missed false alarm object in any of the frame images; otherwise, it is determined that there is a missed false alarm object in any of the frame images.

11. The device according to claim 7, characterized in that The acquisition submodule is specifically used for: Acquire from the video stream an image region whose acquisition time is within the time range, whose position is within the at least one space range and which is a foreground, so as to obtain a plurality of image regions; Acquire from the video stream an image whose acquisition time is within the time range and does not have a foreground, so as to obtain a plurality of third images; Clustering the plurality of image regions to obtain a plurality of first clustering results, and clustering the plurality of third images to obtain a plurality of second clustering results; A plurality of image regions are selected from the plurality of first clustering results as the plurality of foreground images, and a plurality of third images are selected from the plurality of second clustering results as the plurality of background images.

12. The device according to claim 7, characterized in that The fusion module is specifically used for: Determine the acquisition time of a frame of image where the multiple foreground images are located in the video stream, and the positions of the multiple foreground images in the frame of image where the multiple foreground images are located, so as to obtain the acquisition time and image positions corresponding to the multiple foreground images respectively; Performing semantic segmentation on the multiple background images respectively to determine semantic segmentation information corresponding to the multiple background images respectively; Determine the acquisition time of a frame of image where the multiple background images are located in the video stream to obtain the acquisition time corresponding to the multiple background images respectively; The multiple foreground images and the multiple background images are fused based on the acquisition time and image positions respectively corresponding to the multiple foreground images and the acquisition time and semantic segmentation information respectively corresponding to the multiple background images.

13. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any one of the methods described in claims 1-6.

14. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images