Object position generation method and apparatus, electronic device, and computer readable medium

Abstract

Embodiments of the present disclosure disclose an object position generation method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises: obtaining a video captured by a camera in a preset time period as an initial video; determining each frame of image in the initial video as an initial video frame to obtain an initial video frame sequence; for an initial video frame in the initial video frame sequence, the following processing steps are performed: in response to determining that the initial video frame is not a terminal initial video frame, performing data enhancement processing on the initial video frame to generate an enhanced image; in response to determining that the next initial video frame is a terminal initial video frame, determining at least one target object position information of the next initial video frame as the at least one target object position information of the terminal initial video frame. This embodiment realizes object position generation, strengthens feature extraction, and improves generalization ability.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more particularly to object location generation methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] Object location generation is a technique for tracking small targets in complex scenes. Currently, object location generation works by extracting features of the target object using target tracking algorithms such as Bayesian filtering, and then using metrics such as Mahalanobis distance to determine a similarity measure in order to generate the object's location.

[0003] However, when using the above method, the following technical problems often arise:

[0004] First, as more and more video frames are processed, feature extraction becomes insufficient, resulting in poor generalization ability.

[0005] Second, during the object location generation process, errors may occur in the probability distribution network model filtering of false detection information, leading to inaccurate object location generation.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure provide object location generation methods, apparatuses, electronic devices, and computer-readable media to address one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide a method for generating object locations. The method includes: acquiring video captured by a camera within a preset time period as an initial video; determining each frame of the initial video as an initial video frame to obtain an initial video frame sequence; for each initial video frame in the initial video frame sequence, performing the following processing steps: in response to determining that the initial video frame is not a terminating initial video frame, performing data augmentation processing on the initial video frame to generate an augmented image; inputting the augmented image into a target detection network model to obtain at least one target object detection information; inputting the augmented image into a target location probability distribution network model to obtain at least one target object probability distribution information; generating at least one target object location information of the initial video frame based on the at least one target object detection information and the at least one target object probability distribution information; generating at least one target object location information in the next initial video frame based on the at least one target object location information of the initial video frame; in response to determining that the next initial video frame is a terminating initial video frame, determining that at least one target object location information of the next initial video frame is at least one target object location information of the terminating initial video frame, wherein the terminating initial video frame is the initial video frame located at the end frame position in the initial video frame sequence.

[0010] Secondly, some embodiments of this disclosure provide an object location generation apparatus, comprising: an acquisition unit configured to acquire video captured by a camera within a preset time period as an initial video; a determination unit configured to determine each frame of the initial video as an initial video frame, thereby obtaining an initial video frame sequence; and a processing unit configured to perform the following processing steps on the initial video frames in the initial video frame sequence: in response to determining that the initial video frame is not a terminated initial video frame, performing data augmentation processing on the initial video frame to generate an augmented image; inputting the augmented image into a target detection network model to obtain at least one target object detection information; and inputting the augmented image into... The network model of target location probability distribution is used to obtain at least one target object probability distribution information; based on at least one target object detection information and at least one target object probability distribution information, at least one target object location information of the initial video frame is generated; based on at least one target object location information of the initial video frame, at least one target object location information in the next initial video frame is generated; in response to determining that the next initial video frame is the terminating initial video frame, at least one target object location information of the next initial video frame is determined to be at least one target object location information of the terminating initial video frame, wherein the terminating initial video frame is the initial video frame located at the end frame position in the initial video frame sequence.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0013] The above embodiments of this disclosure have the following beneficial effects: the object location generation method of some embodiments of this disclosure strengthens feature extraction and improves generalization ability. Specifically, the reason for insufficient feature extraction and poor generalization ability is that as more and more video frames are processed, feature extraction becomes insufficient and generalization ability is poor. Based on this, the object location generation method of some embodiments of this disclosure first acquires video captured by a camera within a preset time period as initial video. Thus, a video for subsequent processing can be obtained. Next, each frame image in the above initial video is determined as an initial video frame, resulting in an initial video frame sequence. Thus, each frame image of the video can be obtained. Then, for the initial video frames in the above initial video frame sequence, the following processing steps are performed: First, in response to determining that the initial video frame is not a terminated initial video frame, data augmentation processing is performed on the initial video frame to generate an enhanced image. Thus, data augmentation processing is performed on the initial video frames in the initial video frame sequence, making the features of the initial video frames richer. Second, the enhanced image is input into an object detection network model to obtain at least one target object detection information. Therefore, the enhanced image can be target-detected using an object detection network model, improving the accuracy of target object detection and obtaining at least one target object detection information, thus enhancing feature extraction. The third step involves inputting the enhanced image into a target location probability distribution network model to obtain at least one target object probability distribution information. This allows the enhanced image to be predicted using the target location probability distribution network model, obtaining the probability distribution of at least one target object in the enhanced image, thereby reducing the probability of false detections. The fourth step generates at least one target object location information for the initial video frame based on the at least one target object detection information and the at least one target object probability distribution information. This allows the at least one target object detection information and the at least one target object probability distribution information to be combined, thus using the combined information as the at least one target object location information for the initial video frame. The fifth step generates at least one target object location information for the next initial video frame based on the at least one target object location information from the initial video frame. This yields at least one target object location information for the next frame. Step 6: In response to determining that the next initial video frame is the terminating initial video frame, at least one target object location information of the next initial video frame is determined to be at least one target object location information of the terminating initial video frame, wherein the terminating initial video frame is the initial video frame located at the end frame position in the initial video frame sequence. This enhances feature extraction and, consequently, improves generalization ability. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 This is a flowchart of some embodiments of the object location generation method according to this disclosure;

[0016] Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the object location generation apparatus according to this disclosure;

[0017] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Figure 1This is a flow 100 of some embodiments of the object location generation method disclosed herein. The object location generation method includes the following steps:

[0025] Step 101: Obtain the video captured by the camera within a preset time period as the initial video.

[0026] In some embodiments, the execution subject of the object location generation method (e.g., a computing device) can acquire video captured by a camera within a preset time period via a wired or wireless connection, as the initial video. The preset time period can refer to a pre-defined time range. For example, the preset time period could be from 14:00:00 on June 20, 2023 to 12:00:00 on June 21, 2023. The initial video can include an initial video consisting of at least two video frames. For example, the initial video could refer to video captured by the camera of a road segment including pedestrians, trees, small animals, etc. It should be noted that the wireless connection method can include, but is not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future wireless connection methods.

[0027] Step 102: Determine each frame of the above initial video as an initial video frame to obtain an initial video frame sequence.

[0028] In some embodiments, the execution entity may determine each frame of the initial video as an initial video frame to obtain an initial video frame sequence.

[0029] As an example, the aforementioned executing entity can determine each frame of the initial video as an initial video frame, and determine the determined initial video frames as an initial video frame sequence.

[0030] Step 103: For the initial video frames in the above initial video frame sequence, perform the following processing steps:

[0031] Step 1031: In response to determining that the initial video frame is not the terminating initial video frame, data augmentation processing is performed on the initial video frame to generate an enhanced image.

[0032] In some embodiments, the execution entity may perform data augmentation processing on the initial video frame in response to determining that the initial video frame is not a terminated initial video frame, to generate an enhanced image. Here, the data augmentation processing may refer to image noise reduction processing. Here, the enhanced image is an image that more clearly represents the image features of the initial video frame.

[0033] Step 1032: Input the enhanced image into the target detection network model to obtain at least one target object detection information.

[0034] In some embodiments, the execution entity may input the enhanced image into an object detection network model to obtain at least one object detection information. Here, the object detection network model is a network model used to detect objects in the enhanced image. The object detection network model includes a candidate box extraction network, which is used to extract candidate boxes for objects in the enhanced image, and includes at least one convolutional layer, at least one activation function layer, and at least one pooling layer. Here, the at least one object detection information may refer to the location information of at least one detected object.

[0035] Optionally, the aforementioned execution entity can input the enhanced image into the object detection network model through the following steps to obtain at least one object detection information:

[0036] The first step is to input the enhanced image into a candidate bounding box extraction network to obtain at least one candidate bounding box. Here, the candidate bounding box information in the above-mentioned at least one candidate bounding box information can refer to the coordinates of the candidate bounding box.

[0037] The second step involves performing the following processing steps on at least one of the candidate box information mentioned above:

[0038] Sub-step one involves performing convolution processing on each candidate box from the at least one candidate box information to generate candidate offset information, thus obtaining a candidate offset information set. Here, convolution processing can refer to grouped convolution. Here, the candidate offset information can include horizontal and vertical offsets. Here, the horizontal offset can refer to the x-coordinate of the offset. Here, the horizontal offset can refer to the y-coordinate of the offset. Here, the offsets can be used to characterize the offset of the candidate box. For example, the candidate offset information can be a tuple (x, y), where x can refer to the horizontal offset and y can refer to the vertical offset.

[0039] Sub-step two involves performing bounding box regression processing on each candidate offset information in the aforementioned candidate offset information set to generate target candidate offset information, thus obtaining the target candidate offset information set. Here, the aforementioned target candidate offset information can refer to the offset information of candidate boxes that are closer to the actual object.

[0040] As an example, the aforementioned execution entity may first adjust the position of each candidate offset information in the aforementioned candidate offset information set, and then use the position-adjusted candidate offset information as the target candidate offset information.

[0041] Sub-step three involves generating target candidate bounding box information corresponding to each target candidate offset information included in the aforementioned target candidate offset information set, thus obtaining a target candidate bounding box information set. Here, the aforementioned target candidate bounding box information may refer to the coordinate information of the object corresponding to the aforementioned target candidate offset information.

[0042] Sub-step four involves performing non-maximum suppression on each target candidate box in the aforementioned target candidate box information set to generate filtered candidate boxes, resulting in a filtered candidate box set. Here, the filtered candidate boxes can represent the candidate boxes after removing redundant candidate boxes from the target candidate box information.

[0043] Sub-step five: Based on the above-mentioned candidate box set after filtering and the corresponding target candidate offset information set, determine at least one target object detection information.

[0044] As an example, the aforementioned execution entity can first combine the filtered candidate boxes in the above-mentioned filtered candidate box set with the target candidate offset information in the corresponding target candidate offset information set to obtain the combined information. Then, the combined information is used as the target object detection information to obtain at least one target object detection information.

[0045] Optionally, the above object detection network model can be trained through the following steps:

[0046] The first step is to obtain a sample set. This sample set includes enhanced video frame samples and corresponding enhanced video frame sample target information.

[0047] The second step involves performing the following training steps based on the sample set:

[0048] Sub-step one involves inputting the enhanced video frame samples of at least one sample in the sample set into the initial target detection network model to obtain the enhanced video frame sample prediction information corresponding to each of the at least one sample.

[0049] Sub-step two involves comparing the enhanced video frame sample prediction information corresponding to each of the at least one sample with the corresponding enhanced video frame sample target information. In practice, the execution entity can use the cross-entropy loss function to compare and determine the gap between the enhanced video frame sample prediction information and the corresponding enhanced video frame sample target information corresponding to each of the at least one sample.

[0050] Sub-step three involves determining whether the initial object detection network model has achieved the preset optimization objective based on the comparison results. This optimization objective can be achieving a preset number of training rounds or iterations.

[0051] As an example, the aforementioned execution entity can first determine the similarity between the enhanced video frame sample prediction information and the corresponding enhanced video frame sample target information for each sample in at least one sample, thereby obtaining a similarity set. Then, the average similarity of the similarity in the similarity set is calculated to obtain the average similarity. Finally, the average similarity is used to characterize whether the initial target detection network model has achieved the preset optimization objective.

[0052] Sub-step four: In response to determining that the initial object detection network model has achieved the above optimization objective, the initial object detection network model is used as the trained object detection network model.

[0053] The third step, in response to the determination that the initial object detection network model has not achieved the above optimization objective, is to adjust the network parameters of the initial object detection network model, and to use unused samples to form a sample set. The adjusted initial object detection network model is then used as the new initial object detection network model, and the above training steps are executed again. As an example, the back propagation algorithm (BP algorithm) and gradient descent methods (such as mini-batch gradient descent) can be used to adjust the network parameters of the initial object detection network model.

[0054] Step 1033: Input the enhanced image into the target location probability distribution network model to obtain at least one target object probability distribution information.

[0055] In some embodiments, the execution entity can input the enhanced image into a target location probability distribution network model to obtain probability distribution information of at least one target object. Here, the target location probability distribution network model is a network model used to obtain the probability distribution of object positions in the enhanced image. The target location probability distribution network model includes a probability prediction distribution network and a similarity neural network, wherein the probability prediction distribution network is used to obtain the probability distribution of object positions and includes an input layer, hidden layers, a summing layer, and an output layer. The similarity neural network can refer to a Siamese neural network. Here, the probability distribution information of at least one target object can refer to the probability distribution values ​​of at least one detected object.

[0056] Optionally, the aforementioned execution entity can input the enhanced image into the target location probability distribution network model through the following steps to obtain at least one target object probability distribution information:

[0057] The first step involves inputting the enhanced image into the probability prediction distribution network to obtain the probability distribution of at least one target object for the enhanced image, which serves as an image probability distribution set. Here, the image probability distribution in the image probability distribution set is the probability distribution of the target object within the probability distribution of at least one target object. The probability distribution of at least one target object can refer to the position probability distribution value of at least one object. For example, the probability distribution could refer to 30%.

[0058] The second step involves performing state estimation on the enhanced image using a particle filtering algorithm to obtain a set of state particles. Here, the particle filtering algorithm is an algorithm used to track target objects. The state particles in the state particle set can refer to samples propagating in the state space. The state estimation can refer to predicting the state of each particle.

[0059] The third step involves inputting the aforementioned set of state particles into a probability prediction distribution network to obtain the particle probability set corresponding to the set of state particles. Here, the particle probability in the aforementioned particle probability set can refer to the probability distribution of each particle approaching the target object.

[0060] The fourth step involves comparing each state particle in the aforementioned state particle set with the enhanced image to generate a difference value corresponding to each state particle, thus obtaining a difference value set. Here, the comparison can refer to comparison using the cross-entropy loss function. The difference value in the difference value set can refer to the gap between the enhanced image and each state particle.

[0061] The fifth step is to generate a weight for each state particle based on the above difference set, thus obtaining a weight set. In practice, first, the mean of the above difference set is determined to obtain the average difference, and then the ratio of each state particle to the average difference is determined as the weight, thus obtaining the weight set.

[0062] Step 6: For each state particle in the aforementioned set of state particles, perform a weighted summation of the weight corresponding to the state particle with the probability of each particle in the aforementioned particle probability set to generate a particle prediction probability set. Here, the weighted summation process can refer to multiplying the weight corresponding to the state particle with the probability of each particle in the aforementioned particle probability set and then summing the results. The aforementioned particle prediction probability can represent the probability distribution of each state particle.

[0063] Step 7: For each particle prediction probability in the aforementioned particle prediction probability set, input the aforementioned particle prediction probability and each image probability distribution in the aforementioned image probability distribution set into a similarity neural network to obtain a similarity set as a probability distribution score set. Here, the probability distribution score in the aforementioned probability distribution score set can refer to the difference between the aforementioned particle prediction probability and each image probability distribution in the aforementioned image probability distribution set.

[0064] Step 8: Sort the above probability distribution score set to generate a probability distribution score sequence. Here, sorting can refer to ranking the probability distribution score set from smallest to largest. The above probability distribution score sequence is the sorted probability distribution score sequence.

[0065] The ninth step involves identifying the image probability distributions in the probability distribution score sequence that correspond to images with scores greater than a preset score as at least one target object probability distribution. Here, the setting of the preset score is not restricted.

[0066] The relevant content in steps one through nine above constitutes an inventive point of this disclosure, solving the second technical problem mentioned in the background art: "inaccurate object location generation." Factors leading to inaccurate object location generation often include: errors in filtering false detection information by the probability distribution network model during object location generation, resulting in inaccurate object location generation. Solving these factors can improve the accuracy of object location generation. To achieve this, firstly, the enhanced image is input into the probability prediction distribution network to obtain the probability distribution of at least one target object in the enhanced image as an image probability distribution set. Thus, the probability distribution set of at least one target object in the enhanced image can be obtained through the probability prediction distribution network. Next, the enhanced image is state-estimated using a particle filtering algorithm to obtain a state particle set. Thus, the state particle set corresponding to the enhanced image can be obtained. Then, the state particle set is input into the probability prediction distribution network to obtain the particle probability set corresponding to the state particle set. Thus, the particle probability set corresponding to the state particle set can be obtained through the probability prediction distribution network. Next, each state particle in the aforementioned state particle set is compared with the enhanced image to generate a difference value for each state particle, resulting in a difference value set. This reduces the possibility of false detection errors that might occur during object location generation due to the probability distribution network model filtering out false positives. Then, based on the difference value set, weights are generated for each state particle, resulting in a weight set. This provides a weight set for subsequent operations. Next, for each state particle in the aforementioned state particle set, the weights corresponding to the state particle are weighted and summed with the probabilities of each particle in the aforementioned particle probability set to generate a particle prediction probability set. This improves the accuracy of object location generation. Then, for each particle prediction probability in the aforementioned particle prediction probability set, the particle prediction probability and the image probability distribution in the aforementioned image probability distribution set are input into a similarity neural network to obtain a similarity set as a probability distribution score set. This yields a probability distribution score set corresponding to the particle prediction probability and the image probability distribution in the aforementioned image probability distribution set. Finally, the probability distribution score set is sorted to generate a probability distribution score sequence. Therefore, the probability distribution score set can be sorted from smallest to largest to obtain a probability distribution score sequence. Finally, the image probability distributions in the probability distribution set corresponding to scores greater than a preset score in the above probability distribution score sequence are determined as at least one target object probability distribution information. This reduces the possibility of false detection errors that may occur during the object location generation process due to the probability distribution network model filtering false detection information. Consequently, the accuracy of object location generation is improved.

[0067] Optionally, the above target location probability distribution network model is trained through the following steps:

[0068] The first step is to obtain a sample set of enhanced video frames. Here, obtaining a sample set of enhanced video frames means randomly acquiring such a set. Specifically, the enhanced video frame samples in the sample set refer to video frames obtained by enhancing the video captured by the camera within a preset time period and then segmenting them into frames.

[0069] The second step is to select enhanced video frame samples from the aforementioned enhanced video frame sample set as enhanced video frame test samples. These enhanced video frame test samples include: the enhanced video frame sample and its corresponding target object probability distribution prediction information label. Here, the selection can refer to random selection.

[0070] The third step involves inputting the enhanced video frame test samples into the initial target location probability distribution network model to obtain the target object probability distribution prediction information. Here, the initial target location probability distribution network model can be an untrained target location probability distribution network model.

[0071] The fourth step involves determining the label difference between the predicted probability distribution information of the target object and the predicted probability distribution information of the target object included in the enhanced video frame test samples, based on a preset distribution information loss function. Here, the distribution information loss function may include, but is not limited to, mean squared error (MSE) loss function, cross-entropy loss function, etc.

[0072] Fifth step: In response to the above label difference value being greater than or equal to the preset label threshold, adjust the network parameters of the above initial target location probability distribution network model.

[0073] In some embodiments, the execution entity may adjust the network parameters of the initial target position probability distribution network model in response to the difference value being greater than or equal to a preset label threshold. Here, the setting of the preset label threshold is not limited. For example, the difference between the difference value and the preset label threshold can be calculated to obtain the loss difference. Based on this, methods such as backpropagation and stochastic gradient descent are used to propagate the difference value from the last layer of the model forward to adjust the parameters of each layer. Of course, as needed, the network dropout method can also be used to keep the network parameters of some layers unchanged without adjustment; no limitations are imposed on this.

[0074] Step 6: In response to the fact that the above label difference value is less than the above preset label threshold, the above initial target location probability distribution network model is determined as the target location probability distribution network model.

[0075] In some embodiments, the execution entity may determine the initial target location probability distribution network model as the target location probability distribution network model in response to the label difference value being less than the preset label threshold. Here, there is no limitation on the setting of the preset label threshold.

[0076] Step 1034: Generate at least one target object location information for the initial video frame based on at least one target object detection information and at least one target object probability distribution information.

[0077] In some embodiments, the execution entity may generate at least one target object location information for an initial video frame based on at least one target object detection information and at least one target object probability distribution information. Here, the target object location information in the at least one target object location information may refer to the coordinate information of the object.

[0078] As an example, at least one target object detection information is combined with at least one target object probability distribution information to obtain at least one target object location information in the initial video frame.

[0079] Step 1035: Based on the position information of at least one target object in the first initial video frame, generate position information of at least one target object in the next initial video frame.

[0080] In some embodiments, the execution entity may generate at least one target object location information in the next initial video frame based on at least one target object location information in the initial video frame.

[0081] As an example, the aforementioned execution entity can determine the similarity of at least one target object location information in the initial video frame to obtain a similarity value, and then predict the location information of at least one target object in the next initial video frame based on the similarity value.

[0082] Step 1036: In response to determining that the next initial video frame is a termination initial video frame, determine that at least one target object position information of the next initial video frame is at least one target object position information of the termination initial video frame.

[0083] In some embodiments, the execution entity may, in response to determining that the next initial video frame is a termination initial video frame, determine at least one target object location information of the next initial video frame as at least one target object location information of the termination initial video frame. The termination initial video frame is the initial video frame located at the end frame position in the initial video frame sequence.

[0084] Optionally, after step 103, the aforementioned executing entity may also perform the following steps:

[0085] In the first step, in response to determining that the next initial video frame is not the termination initial video frame, the next initial video frame corresponding to at least one target object location information is used as the initial video frame, and the above processing steps are executed again.

[0086] The above embodiments of this disclosure have the following beneficial effects: the object location generation method of some embodiments of this disclosure strengthens feature extraction and improves generalization ability. Specifically, the reason for insufficient feature extraction and poor generalization ability is that as more and more video frames are processed, feature extraction becomes insufficient and generalization ability is poor. Based on this, the object location generation method of some embodiments of this disclosure first acquires video captured by a camera within a preset time period as initial video. Thus, a video for subsequent processing can be obtained. Next, each frame image in the above initial video is determined as an initial video frame, resulting in an initial video frame sequence. Thus, each frame image of the video can be obtained. Then, for the initial video frames in the above initial video frame sequence, the following processing steps are performed: First, in response to determining that the initial video frame is not a terminated initial video frame, data augmentation processing is performed on the initial video frame to generate an enhanced image. Thus, data augmentation processing is performed on the initial video frames in the initial video frame sequence, making the features of the initial video frames richer. Second, the enhanced image is input into an object detection network model to obtain at least one target object detection information. Therefore, the enhanced image can be target-detected using an object detection network model, improving the accuracy of target object detection, obtaining at least one target object detection information, and enhancing feature extraction. The third step involves inputting the enhanced image into a target location probability distribution network model to obtain at least one target object probability distribution information. This allows the enhanced image to be predicted using the target location probability distribution network model, obtaining the probability distribution of at least one target object in the enhanced image, thereby reducing the probability of false detections. The fourth step generates at least one target object location information for the initial video frame based on the at least one target object detection information and the at least one target object probability distribution information. This allows the at least one target object detection information and the at least one target object probability distribution information to be combined, thus using the combined information as the at least one target object location information for the initial video frame. The fifth step generates at least one target object location information for the next initial video frame based on the at least one target object location information from the initial video frame. This allows the at least one target object location information for the next frame to be obtained. Step 6: In response to determining that the next initial video frame is the terminating initial video frame, at least one target object location information of the next initial video frame is determined to be at least one target object location information of the terminating initial video frame, wherein the terminating initial video frame is the initial video frame located at the end frame position in the initial video frame sequence. This enhances feature extraction and, consequently, improves generalization ability.

[0087] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of the object location generation method, and these apparatus embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0088] like Figure 2 As shown, an operator-based image detection device 200 in some embodiments includes: an acquisition unit 201, a determination unit 202, and a processing unit 203. The acquisition unit 201 is configured to acquire video captured by a camera within a preset time period as initial video; the determination unit 202 is configured to determine each frame in the initial video as an initial video frame, obtaining an initial video frame sequence; the processing unit 203 is configured to perform the following processing steps on the initial video frames in the initial video frame sequence: in response to determining that the initial video frame is not a termination initial video frame, perform data augmentation processing on the initial video frame to generate an enhanced image; input the enhanced image into a target detection network model to obtain at least one target object detection information; and input the enhanced image into a target location probability distribution network. The model obtains at least one target object probability distribution information; based on at least one target object detection information and at least one target object probability distribution information, at least one target object position information of an initial video frame is generated; based on at least one target object position information of the initial video frame, at least one target object position information in the next initial video frame is generated; in response to determining that the next initial video frame is a terminating initial video frame, at least one target object position information of the next initial video frame is determined to be at least one target object position information of the terminating initial video frame, wherein the terminating initial video frame is the initial video frame located at the end frame position in the initial video frame sequence.

[0089] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0090] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0091] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 304. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 304 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0092] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0093] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0094] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0095] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0096] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire video captured by a camera within a preset time period as initial video; determine each frame of the initial video as an initial video frame, obtaining an initial video frame sequence; for the initial video frames in the initial video frame sequence, perform the following processing steps: in response to determining that the initial video frame is not a terminated initial video frame, perform data augmentation processing on the initial video frame to generate an augmented image; input the augmented image into a target detection network model to obtain at least one target object detection information; input the augmented image to the target location. A probability distribution network model is used to obtain probability distribution information of at least one target object; based on the detection information of at least one target object and the probability distribution information of at least one target object, at least one target object position information of an initial video frame is generated; based on the at least one target object position information of the initial video frame, at least one target object position information in the next initial video frame is generated; in response to determining that the next initial video frame is a terminating initial video frame, at least one target object position information of the next initial video frame is determined to be at least one target object position information of the terminating initial video frame, wherein the terminating initial video frame is the initial video frame located at the end frame position in the initial video frame sequence.

[0097] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The units described in some embodiments of this disclosure can be implemented in software or hardware. The units described above can also be housed in a processor; for example, a processor can be described as including: an acquisition unit, a determination unit, and a processing unit. The names of these units do not necessarily limit the unit itself; for example, the acquisition unit can also be described as "a unit that acquires video captured by a camera within a preset time period as initial video."

[0100] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0101] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating object location, comprising: Acquire videos captured by the camera within a preset time period as the initial video; Each frame in the initial video is determined as the initial video frame, resulting in an initial video frame sequence; For the initial video frame in the initial video frame sequence, perform the following processing steps: In response to determining that the initial video frame is not the terminating initial video frame, data augmentation is performed on the initial video frame to generate an enhanced image; The enhanced image is input into the object detection network model to obtain at least one object detection information. The enhanced image is input into the target location probability distribution network model to obtain at least one target object probability distribution information, including: The enhanced image is input into the probability prediction distribution network to obtain the probability distribution of at least one target object in the enhanced image as an image probability distribution set; The enhanced image is state estimated using a particle filtering algorithm to obtain a state particle set. The state particle set is then input into a probability prediction distribution network to obtain a particle probability set corresponding to the state particle set. Each state particle in the state particle set is compared with the enhanced image to generate a difference value corresponding to each state particle, thus obtaining a difference value set; Based on the difference set, a weight corresponding to each state particle is generated, resulting in a weight set; For each state particle in the state particle set, the weight corresponding to the state particle is weighted and summed with the probability of each particle in the particle probability set to generate the particle prediction probability, thus obtaining the particle prediction probability set. For each particle prediction probability in the particle prediction probability set, the particle prediction probability and each image probability distribution in the image probability distribution set are input into a similarity neural network to obtain a similarity set as a probability distribution score set. The probability distribution score set is sorted to generate a probability distribution score sequence. The image probability distribution in the probability distribution score sequence that is greater than a preset score is determined as at least one target object probability distribution information. Based on at least one target object detection information and at least one target object probability distribution information, generate at least one target object location information for the initial video frame; Based on the location information of at least one target object in the initial video frame, generate the location information of at least one target object in the next initial video frame. In response to determining that the next initial video frame is a termination initial video frame, at least one target object position information of the next initial video frame is determined to be at least one target object position information of the termination initial video frame, wherein the termination initial video frame is the initial video frame located at the end frame position in the initial video frame sequence.

2. The method according to claim 1, wherein, The method further includes: In response to determining that the next initial video frame is not the termination initial video frame, the next initial video frame corresponding to at least one target object location information is used as the initial video frame, and the processing steps are executed again.

3. The method according to claim 1, wherein, The object detection network model is trained through the following steps: Obtain a sample set, wherein the samples in the sample set include enhanced video frame samples and enhanced video frame sample target information corresponding to the enhanced video frame samples; Perform the following training steps based on the sample set: The enhanced video frame samples of at least one sample in the sample set are input into the initial target detection network model to obtain the enhanced video frame sample prediction information corresponding to each of the at least one sample. The enhanced video frame sample prediction information corresponding to each of the at least one sample is compared with the corresponding enhanced video frame sample target information. Based on the comparison results, determine whether the initial target detection network model has achieved the preset optimization objective; In response to determining that the initial object detection network model has reached the optimization objective, the initial object detection network model is used as the trained object detection network model. In response to the determination that the initial object detection network model has not reached the optimization objective, the network parameters of the initial object detection network model are adjusted, and a sample set is formed using unused samples. The adjusted initial object detection network model is then used as the initial object detection network model, and the training steps are performed again.

4. The method according to claim 1, wherein, The target location probability distribution network model is trained through the following steps: Obtain the enhanced video frame sample set; Enhanced video frame samples are selected from the enhanced video frame sample set as enhanced video frame test samples, wherein the enhanced video frame test samples include: sample enhanced video frames and corresponding target object probability distribution prediction information labels. The enhanced video frame test samples are input into the initial target location probability distribution network model to obtain the target object probability distribution prediction information; Based on a preset distribution information loss function, the label difference value between the target object probability distribution prediction information and the target object probability distribution prediction information labels included in the enhanced video frame test samples is determined. In response to the label difference value being greater than or equal to a preset label threshold, the network parameters of the initial target location probability distribution network model are adjusted.

5. The method according to claim 4, wherein, The method further includes: In response to the label difference value being less than the preset label threshold, the initial target location probability distribution network model is determined as the target location probability distribution network model.

6. The method according to claim 1, wherein, The object detection network model includes: a candidate box extraction network; and The step of inputting the enhanced image into the target detection network model to obtain at least one target object detection information includes: The enhanced image is input into a candidate box extraction network to obtain at least one candidate box information; The following processing steps are performed on the at least one candidate box information: Each candidate box in the at least one candidate box information is convolved to generate candidate offset information, thus obtaining a candidate offset information set; Bounding box regression processing is performed on each candidate offset information in the candidate offset information set to generate target candidate offset information, thus obtaining the target candidate offset information set; Based on each target candidate offset information included in the target candidate offset information set, target candidate box information corresponding to the target candidate offset information is generated to obtain the target candidate box information set; Non-maximum suppression is applied to each target candidate box in the target candidate box information set to generate filtered candidate boxes, thus obtaining a filtered candidate box set; Based on the filtered candidate box set and the corresponding target candidate offset information set, at least one target object detection information is determined.

7. An object location generation apparatus, comprising: The acquisition unit is configured to acquire video footage captured by the camera within a preset time period as the initial video. The determining unit is configured to determine each frame of the initial video as an initial video frame, thereby obtaining an initial video frame sequence; The processing unit is configured to perform the following processing steps for an initial video frame in the initial video frame sequence: in response to determining that the initial video frame is not a terminating initial video frame, perform data augmentation processing on the initial video frame to generate an enhanced image; The enhanced image is input into the object detection network model to obtain at least one object detection information. The process involves inputting the enhanced image into a target location probability distribution network model to obtain probability distribution information for at least one target object. This includes: inputting the enhanced image into the probability prediction distribution network to obtain a probability distribution of at least one target object for the enhanced image as an image probability distribution set; performing state estimation on the enhanced image using a particle filtering algorithm to obtain a state particle set; inputting the state particle set into the probability prediction distribution network to obtain a particle probability set corresponding to the state particle set; comparing each state particle in the state particle set with the enhanced image to generate a difference value for each state particle, obtaining a difference value set; generating a weight for each state particle based on the difference value set, obtaining a weight set; for each state particle in the state particle set, performing a weighted summation of the weight corresponding to the state particle with the probability of each particle in the particle probability set to generate a particle prediction probability set; and performing a weighted summation of the predicted probability of each particle in the particle prediction probability set. The probability distribution involves inputting the particle prediction probability and the probability distribution of each image in the image probability distribution set into a similarity neural network to obtain a similarity set as a probability distribution score set. The probability distribution score set is then sorted to generate a probability distribution score sequence. Image probability distributions in the probability distribution score sequence that correspond to a preset score are identified as at least one target object probability distribution information. Based on at least one target object detection information and at least one target object probability distribution information, at least one target object position information for an initial video frame is generated. Based on the at least one target object position information for the initial video frame, at least one target object position information for the next initial video frame is generated. In response to determining that the next initial video frame is a terminating initial video frame, at least one target object position information for the next initial video frame is determined as at least one target object position information for the terminating initial video frame, wherein the terminating initial video frame is the initial video frame located at the end frame position in the initial video frame sequence.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images