Training sample set generation method and device, electronic equipment and computer readable medium
By obtaining the frame image sequence and target proofing images of the target video, the training sample set is generated using feature information matching, which solves the problems of manual labeling errors and motion uncertainty, improves image quality and efficiency, and realizes high-quality training sample set generation .
Patent Information
- Application Number
- CN202411054446.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, errors are prone to occur when manually labeling image samples, and different labelers understand differently, resulting in inconsistent labeling and high labor and time costs; uncertainty in the motion of the target object in the frame image sequence leads to large errors in the detection result; the target object Size changes and lighting changes lead to blurring of edges.
By acquiring the frame image sequence of the target video and the target proof image, the initial and terminated sample frame images are generated using feature information matching, a candidate sample set is formed, and a training sample set is generated based on the image similarity, redundant images are removed, and image quality is improved.
It realizes high-quality training sample set generation, saves manpower and time costs, improves the accuracy and efficiency of image annotation, reduces redundant images, and reduces storage costs.
Smart Images

Figure CN118968220A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a training sample set generation method, device, electronic device, and computer-readable medium. Background Art
[0002] At present, data is one of the three elements of artificial intelligence, and excellent training data, especially manually labeled training data, is an important guarantee for the effectiveness of algorithm models. When obtaining image samples of target objects, in order to ensure the effectiveness of related detection and segmentation algorithms, it is often necessary to annotate images of the same target object from different perspectives, and take into account the lighting conditions of the target object, background changes, etc. For the annotation of image samples, the usual method is to manually annotate the position and outline of the target object in the image, and take into account factors such as the different sizes, lighting conditions, and background changes of the target object.
[0003] However, when the above method is used to annotate the target object in the image, the following technical problems often occur:
[0004] First, manual annotation is prone to errors, and different annotators may have different understandings and annotation methods for the same image, resulting in inconsistent annotations. In addition, manual annotation is slow, resulting in a significant increase in labor and time costs.
[0005] Second, when labeling target objects in a frame image sequence, it is often necessary to detect the approximate area where the target object is located in the next frame based on the area where the target object is located in the current frame. The uncertainty of the target object's movement will cause large errors in the detection results.
[0006] Third, when annotating images of the same target object from different perspectives, the edge of the target object becomes blurred due to changes in the size of the target object and changes in lighting conditions.
[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention
[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.
[0009] Some embodiments of the present disclosure propose a training sample set generation method, apparatus, electronic device, and computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.
[0010] In a first aspect, some embodiments of the present disclosure provide a method for generating a training sample set, comprising: obtaining a frame image sequence and a target proofing image corresponding to a target video, wherein the target proofing image is a frame image in the target video that contains object feature information corresponding to a target object; for the target proofing image in the frame image sequence, performing the following first determination step: in response to determining that the target proofing image is not an initial frame image in the frame image sequence, performing the following second determination step: in response to determining that the first frame image contains feature information that matches the object feature information, generating a first sample frame image based on the first frame image; in response to determining that the first frame image is not the initial frame image in the frame image sequence, taking the first frame image as the target proofing image and continuing to perform the first determination step; in response to determining that the first frame image is the initial frame image in the frame image sequence, determining the obtained first sample frame image set as a first candidate sample Frame image set; for the target proofing image in the above frame image sequence, perform the following third determination step: in response to determining that the target proofing image is not the terminating frame image in the above frame image sequence, perform the following fourth determination step: in response to determining that the second frame image has feature information matching the feature information of the object, generate a second sample frame image according to the second frame image; in response to determining that the second frame image is not the terminating frame image in the above frame image sequence, use the second frame image as the target proofing image, and continue to perform the above third determination step; in response to determining that the second frame image is the terminating frame image in the above frame image sequence, determine the obtained second sample frame image set as the second candidate sample frame image set; based on the above first candidate sample frame image set and the above second candidate sample frame image set, generate an initial sample set for the above target object; based on the image similarity between the initial sample in the above initial sample set and the above target proofing image, generate a training sample set.
[0011] In a second aspect, some embodiments of the present disclosure provide a training sample set generation device, comprising: an acquisition unit, configured to acquire a frame image sequence and a target proofing image corresponding to a target video, wherein the target proofing image is a frame image in the target video that has object feature information corresponding to a target object; a first determination unit, configured to perform the following first determination step for the target proofing image in the frame image sequence: in response to determining that the target proofing image is not an initial frame image in the frame image sequence, perform the following second determination step: in response to determining that the first frame image has feature information matching the object feature information, generate a first sample frame image based on the first frame image; in response to determining that the first frame image is not the initial frame image in the frame image sequence, use the first frame image as the target proofing image and continue to perform the first determination step; in response to determining that the first frame image is the initial frame image in the frame image sequence, determine the obtained first sample frame image set as a first candidate sample frame image set; The second determination unit is configured to perform the following third determination step for the target proofing image in the above-mentioned frame image sequence: in response to determining that the target proofing image is not the terminating frame image in the above-mentioned frame image sequence, perform the following fourth determination step: in response to determining that the second frame image has feature information matching the feature information of the object, generate a second sample frame image based on the second frame image; in response to determining that the second frame image is not the terminating frame image in the above-mentioned frame image sequence, use the second frame image as the target proofing image and continue to perform the above-mentioned third determination step; in response to determining that the second frame image is the terminating frame image in the above-mentioned frame image sequence, determine the obtained second sample frame image set as the second candidate sample frame image set; the first generation unit is configured to generate an initial sample set for the above-mentioned target object based on the above-mentioned first candidate sample frame image set and the above-mentioned second candidate sample frame image set; the second generation unit is configured to generate a training sample set based on the image similarity between the initial sample in the above-mentioned initial sample set and the above-mentioned target proofing image.
[0012] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.
[0013] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.
[0014] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the training sample set generation method of some embodiments of the present disclosure, a certain frame in the video and the video selected by the target user can be used to automatically generate high-quality training sample images for the target object, effectively saving manpower costs and time costs. Specifically, the reason why the annotation results of the training sample images corresponding to the target object are not accurate is that it is often necessary to manually annotate the position, size and outline of the target object in the image, and it is necessary to take into account factors such as the different sizes of the object, lighting conditions, and background changes. It is difficult to accurately and objectively annotate the target object in the image with manual annotation, which leads to low quality of the training sample images corresponding to the target object. Based on this, the training sample set generation method of some embodiments of the present disclosure first obtains a frame image sequence and a target proofing image corresponding to the target video. Among them, the above-mentioned target proofing image is a frame image in the above-mentioned target video that has object feature information corresponding to the target object. Thus, the target video and the target proofing image selected by the target user can be obtained. Then, for the target proofing image in the above frame image sequence, the following first determination step is performed: In response to determining that the target proofing image is not the initial frame image in the above frame image sequence, the following second determination step is performed: First, in response to determining that the first frame image has feature information matching the object feature information, a first sample frame image is generated according to the first frame image. Thus, by matching the feature information corresponding to the first frame image with the object feature information of the target proofing image, it is determined whether the first frame image contains the object feature information corresponding to the target object. If the first frame image has feature information matching the object feature information, a first sample frame image corresponding to the first frame image is generated. Then, in response to determining that the first frame image is not the initial frame image in the above frame image sequence, the first frame image is used as the target proofing image, and the above first determination step is continued. Thus, by cyclically executing the first determination step, a frame image including the object feature information corresponding to the target object before the initially acquired target proofing image can be obtained more accurately, and a corresponding first sample frame image is generated. Finally, in response to determining that the first frame image is the initial frame image in the above frame image sequence, the obtained first sample frame image set is determined as the first candidate sample frame image set. Thus, all first sample frame images generated by the first determination step can be obtained. Secondly, for the target proofing image in the above frame image sequence, the following third determination step is performed: In response to determining that the target proofing image is not the termination frame image in the above frame image sequence, the following fourth determination step is performed: First, in response to determining that the second frame image has feature information that matches the object feature information, a second sample frame image is generated based on the second frame image. Thus, by matching the feature information corresponding to the second frame image with the target proofing image object feature information, it is determined whether the second frame image contains the object feature information corresponding to the target object.If there is feature information matching the feature information of the object in the second frame image, a first sample frame image corresponding to the second frame image is generated. Then, in response to determining that the second frame image is not the termination frame image in the above frame image sequence, the second frame image is used as the target proofing image, and the above third determination step is continued. Thus, by cyclically executing the third determination step, a frame image including the object feature information corresponding to the target object after the initially acquired target proofing image can be obtained more accurately, and a corresponding second sample frame image is generated. Finally, in response to determining that the second frame image is the termination frame image in the above frame image sequence, the obtained second sample frame image set is determined as the second candidate sample frame image set. Thus, all second sample frame images generated by the third determination step can be obtained. Then, based on the above first candidate sample frame image set and the above second candidate sample frame image set, an initial sample set for the target object is generated. Thus, all frame images in the target video that have the object feature information corresponding to the target object can be obtained. Then, based on the image similarity between the initial sample and the target proofing image in the above initial sample set, a training sample set is generated. Thus, the frame images in the initial sample set that are not much different from the target proofing image can be removed, which reduces the redundant images in the training sample set, improves the quality of the training sample set, and thus reduces the storage cost. In summary, by continuously updating the target proofing image and matching the feature information of the first frame image or the second frame image with the object feature information, it can be more accurately determined whether the first frame image or the second frame image can become a candidate sample frame image. And by removing the redundant images in the training sample set through the image similarity between the initial sample in the initial sample set and the target proofing image, the quality of the training sample set is further improved, effectively saving labor costs and time costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.
[0016] Figure 1 is a flowchart of some embodiments of the training sample set generation method according to the present disclosure;
[0017] Figure 2 is a schematic diagram of the structure of some embodiments of the training sample set generation device according to the present disclosure;
[0018] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION
[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.
[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0025] refer to Figure 1 , shows a process 100 of some embodiments of the training sample set generation method according to the present disclosure. The training sample set generation method comprises the following steps:
[0026] Step 101, obtaining a frame image sequence and a target proofing image corresponding to a target video.
[0027] In some embodiments, the execution subject (e.g., a computing device) of the above-mentioned training sample set generation method can obtain the frame image sequence and the target proofing image corresponding to the target video through a wired connection or a wireless connection. Among them, the above-mentioned target video can be a video for the target object. The above-mentioned target proofing image is a frame image in the above-mentioned target video in which the object feature information corresponding to the target object exists. The above-mentioned target object is an object selected by the target user in the target video and of interest to the target user. The above-mentioned target object can be an item. For example, the above-mentioned item can be a power bank. The above-mentioned object feature information is information used to describe the characteristics of the target object. In practice, the object feature information can be content semantic information that characterizes the feature content of the object. For example, the object feature content can include color information, texture information, and position information of the target object. In practice, first, the target user can browse the target video and randomly select a frame image in the target video that displays the target object as the proofing image. Then, the target user can mark the outline of the target object on the proofing image. Finally, the above-mentioned execution subject can obtain the proofing image annotated by the target user as the target proofing image. In practice, the target user can be notified to adjust the outline of the target object annotated on the target proofing image according to actual needs. Thus, the outline of the marked target object is made more accurate. The execution subject may be a server for managing and storing the target video.
[0028] Step 102: for the target proofing image in the frame image sequence, perform the following first determination step:
[0029] Step 1021, in response to determining that the target proofing image is not an initial frame image in the frame image sequence, performing the following second determination step:
[0030] Step 10211: In response to determining that the first frame image has feature information that matches the feature information of the object, generate a first sample frame image according to the first frame image.
[0031] In some embodiments, the execution subject may generate a first sample frame image based on the first frame image in response to determining that the first frame image has feature information matching the feature information of the object, wherein the first frame image is a frame image that is adjacent to and before the corresponding position of the target proofing image in the frame image sequence.
[0032] In some optional implementations of some embodiments, the execution subject may generate a first sample frame image according to the first frame image in response to determining that the first frame image has feature information matching the feature information of the object through the following steps:
[0033] In the first step, based on the position information corresponding to the target object in the target proofing image, the position information corresponding to the target object in the first frame image is detected to generate first detection position information. The first detection position information may be a set of coordinate points corresponding to the boundary of the first detection area. The first detection area may be an area in which the target object is displayed in the detected first frame image.
[0034] In the second step, the area corresponding to the first detection position information is identified to generate first identification information. The identification method may be a target detection algorithm, such as a YOLOv3 target detection algorithm. The first identification information may be a label used to characterize whether the area corresponding to the first detection position information displays a target object. For example, the first identification information may be "the area corresponding to the first detection position information displays a target object" or "the area corresponding to the first detection position information does not display a target object".
[0035] In the third step, in response to determining that the first identification information indicates that the area corresponding to the first detection position information displays a target object, the area corresponding to the first detection position information is segmented to generate a first circumscribed rectangular frame corresponding to the target object in the first frame image. Among them, the method for segmenting the area corresponding to the first detection position information may be an edge segmentation method. The first circumscribed rectangular frame may be a boundary frame surrounded by the boundary of the area corresponding to the first detection position information. In practice, the boundary of the area corresponding to the first detection position information may be smoothed as needed. The smoothing method may be a Bezier curve fitting method.
[0036] The fourth step is to extract first object feature information corresponding to the target object in the first frame image based on the first circumscribed rectangular frame.
[0037] Step 5: Generate a first sample frame image according to the first object feature information and the first circumscribed rectangular frame. The first object feature information can be used as label information of the first sample frame image. The first sample frame image displays the first circumscribed rectangular frame.
[0038] In some optional implementations of some embodiments, the execution subject may detect the position information corresponding to the target object in the first frame image according to the position information corresponding to the target object in the target proofing image through the following steps to generate first detected position information:
[0039] The first step is to obtain a target time corresponding to a target proofing image in a target video and a first time corresponding to a first frame image in the target video, and determine the absolute value of the difference between the target time and the first time as the scaling time interval.
[0040] The second step is to obtain the initial velocity and initial acceleration corresponding to the target object when it is displayed in the target proofing image according to the frame image sequence. The initial velocity and initial acceleration are the velocity and acceleration corresponding to the target object at the target moment. In practice, the execution subject can use the optical flow method to obtain the initial velocity and initial acceleration.
[0041] The third step is to generate a scaling displacement according to the initial velocity, initial acceleration and scaling time interval. The scaling displacement may be the displacement of the target object within the scaling time interval. The scaling displacement may be [scaling distance, angle]. For example, the scaling displacement may be [1, 90°]. The frame rate corresponding to the target video may be 30 frames per second, and the time interval between two adjacent frame images in the frame image sequence corresponding to the target video is approximately 0.033 seconds. Therefore, within the scaling time interval, the movement of the target object may be approximately regarded as uniformly accelerated linear motion. In practice, the above-mentioned execution subject may use the displacement formula of uniformly accelerated linear motion to determine the scaling displacement.
[0042] The fourth step is to obtain the initial position information and center point coordinates corresponding to the target object in the target proofing image. The initial position information may be a set of coordinate points corresponding to the outline of the target object marked on the target proofing image. The center point coordinates may be the coordinates of the center point of the area corresponding to the initial position information.
[0043] The fifth step is to generate the first target search area corresponding to the first frame image according to the scaling displacement and the center point coordinates. In practice, first, the execution subject may use the position corresponding to the center point coordinates in the first frame image as the starting point of the scaling displacement. Then, the execution subject may determine the end point of the scaling displacement through the starting point of the scaling displacement. For example, the center point coordinates may be (3,4), the scaling displacement may be [1,90°], and the end point of the scaling displacement is (3,5). Finally, the execution subject may determine the area corresponding to the circle with the end point of the scaling displacement as the dot and the preset threshold as the radius as the first target search area.
[0044] The sixth step is to generate an initial state vector according to the initial position information. In practice, the above-mentioned execution entity may determine the coordinate point set corresponding to the initial position information as the target coordinate point set. The above-mentioned execution entity may determine the average value of the sum of the horizontal coordinates of each coordinate point in the target coordinate point set as the initial horizontal coordinate. And the average value of the sum of the vertical coordinates of each coordinate point in the target coordinate point set is determined as the initial vertical coordinate. Thus, the initial coordinates are obtained. In practice, the above-mentioned execution entity may search for the position corresponding to the initial coordinates in the target proofing image, and obtain the initial coordinate velocity corresponding to the initial coordinates at the position at the target moment. Among them, the above-mentioned execution entity may use the optical flow method to obtain the initial coordinate velocity. The initial state vector may be a vector that records the initial coordinates and the initial coordinate velocity.
[0045] Step 7: Establish a state transfer equation. The state transfer equation can be that the state vector at the first moment is equal to the product of the state transfer matrix and the state vector at the target moment and the sum of the noise vector. The state transfer matrix is used to map the state vector of the target object at the target moment to the state vector at the first moment. In practice, the cross-platform computer vision library (Open Source Computer Vision Library, OpenCV) can be used to obtain the noise vector corresponding to the target video.
[0046] Step 8: Generate a first state vector corresponding to the target object in the first frame image according to the state transfer equation and the initial state vector. The first state vector may be the product of the state transfer matrix and the initial state vector and the sum of the noise vector. The first state vector includes the transfer coordinates corresponding to the initial coordinates. The transfer coordinates are the coordinates corresponding to the initial coordinates after the movement of the zoom time interval.
[0047] In the ninth step, a second target search area corresponding to the first frame image is generated according to the first state vector, wherein the second target search area may be an area corresponding to a circle with the transfer coordinates corresponding to the initial coordinates as the center and a preset threshold as the radius.
[0048] In the tenth step, the overlapping area of the first target search area and the second target search area is used as the first detection position area, and first detection position information is generated. The first detection position information may be a set of coordinate points corresponding to the boundary of the first detection position area.
[0049] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology: "When marking the target object in the frame image sequence, it is often necessary to detect the approximate area where the target object is located in the next frame based on the area where the target object is located in the current frame, and the uncertainty of the movement of the target object will cause a large error in the detection result." Based on this, the present disclosure first obtains the target moment corresponding to the target proofing image in the target video and the first moment corresponding to the first frame image in the target video, and determines the absolute value of the difference between the target moment and the first moment as the zoom time interval. Thus, the non-negativity of the time interval is guaranteed. Secondly, the initial velocity and initial acceleration corresponding to the target object in the target proofing image are obtained. According to the initial velocity, initial acceleration and zoom time interval, a zoom displacement is generated. Thus, the displacement of the target object within the zoom time interval is obtained, so that the approximate position of the target object in the first frame image can be obtained by using the zoom displacement. Then, the initial position information and center point coordinates corresponding to the target object in the target proofing image are obtained. According to the zoom displacement and center point coordinates, a first target search area corresponding to the first frame image is generated. Thus, the approximate position of the target object in the first frame image can be obtained. Next, an initial state vector is generated based on the initial position information. Next, a state transfer equation is established. Based on the state transfer equation and the initial state vector, a first state vector corresponding to the target object in the first frame image is generated. Then, based on the first state vector, a second target search area corresponding to the first frame image is generated. Finally, the overlapping area of the first target search area and the second target search area is used as the first detection position area, and the first detection position information is generated. Thus, the overlapping area is used as the first detection position area, so that the range of the first detection position area is more accurate.
[0050] In some optional implementations of some embodiments, the execution subject may identify the area corresponding to the first detection position information through the following steps to generate the first identification information:
[0051] The first step is to segment the first frame image according to the first detection position information to generate a segmented image. In practice, the execution subject may perform edge segmentation on the first frame image to generate a segmented image. The segmented image is an image generated by the area corresponding to the first detection position information.
[0052] The second step is to establish an image enhancement network. The image enhancement network is a neural network that takes the segmented image as input and the enhanced image as output. The image enhancement network includes an input layer, a decomposition layer, a noise reduction layer, an illumination restoration layer and an output layer.
[0053] The third step is to input the segmented image into the input layer of the above-mentioned image enhancement network to generate a preprocessed image corresponding to the segmented image. In practice, first, the above-mentioned execution subject can input the segmented image into the input layer of the image enhancement network. Then, the input layer of the above-mentioned image enhancement network can preprocess the segmented image to generate a preprocessed image. The preprocessing can be filtering the segmented image to improve the quality of the segmented image. Among them, the filtering method can be a non-local mean filtering method.
[0054] The fourth step is to input the preprocessed image into the decomposition layer of the above-mentioned image enhancement network to generate a reflection feature image and an illumination feature image. In practice, first, the above-mentioned execution entity can input the preprocessed image into the decomposition layer of the image enhancement network. Secondly, the decomposition layer of the above-mentioned image enhancement network can perform Gaussian blur processing on the preprocessed image to generate a blurred image. Then, the decomposition layer of the above-mentioned image enhancement network can perform logarithmic operations on the pixel values corresponding to each pixel point in the blurred image to obtain an illumination image. Next, the decomposition layer of the above-mentioned image enhancement network can fuse the preprocessed image with the illumination image to generate a reflection image. The fusion method can be to subtract the pixel values of the pixels at the corresponding positions of the preprocessed image and the illumination image. Finally, the decomposition layer of the above-mentioned image enhancement network can perform a convolution operation on the illumination image and the reflection image to generate an illumination feature image and a reflection feature image. Among them, the convolution operation can be implemented by a convolutional neural network.
[0055] In the fifth step, the illumination feature image is input into the denoising layer of the above-mentioned image enhancement network to generate a denoised feature image. In practice, first, the denoising layer of the above-mentioned image enhancement network can scale the illumination feature image to generate a scaled image set. The scaled image set is an illumination feature image set with different resolutions. Then, the denoising layer of the above-mentioned image enhancement network can perform the following denoising operation on the scaled image set: first, each scaled image in the scaled image set is denoised to generate a denoised image set. The denoising method can be a bilateral filtering method. Then, according to the resolution corresponding to the denoised image, the N denoised images in the denoised image set are respectively determined as the first to N denoised images. The first denoised image has the lowest resolution, and the Nth denoised image has the highest resolution. For example, N can be 3. Finally, the first denoised image is enlarged using an interpolation algorithm. The resolution of the enlarged first denoised image is the same as that of the second denoised image. The enlarged first denoised image is fused with the second denoised image to obtain a first fused image. The fusion method may be to add the pixel values of the pixel points at the corresponding positions of the first denoised image and the second denoised image. The first fused image is enlarged again using the interpolation algorithm. The resolution of the enlarged first fused image is the same as the resolution of the third denoised image. The enlarged first fused image is fused with the third denoised image to obtain a denoised feature image.
[0056] The sixth step is to input the denoised feature image into the illumination restoration layer of the above-mentioned image enhancement network to generate an illumination restoration feature image. In practice, first, the illumination restoration layer of the above-mentioned image enhancement network can transform the denoised feature image from the spatial domain to the frequency domain to obtain the spectrum corresponding to the denoised feature image. Among them, the transformation method can be Fourier transform. Then, the illumination restoration layer of the above-mentioned image enhancement network can filter the high-frequency components in the spectrum to enhance the texture of the denoised feature image. The filtering method can be a high-pass filtering method. Finally, the filtered spectrum is inversely transformed to obtain the illumination restoration feature image.
[0057] In the seventh step, the illumination restoration feature image and the reflection feature image are input to the output layer of the above image enhancement network to generate an enhanced image. In practice, the output layer of the above image enhancement network can use the sum of the pixel values corresponding to the pixel points at the corresponding positions in the illumination restoration feature image and the reflection feature image as the pixel value corresponding to the corresponding pixel point in the enhanced image.
[0058] In the eighth step, the enhanced image is identified to generate first identification information indicating whether the enhanced image displays the target object. The identification method may be a target detection algorithm, such as a YOLOv3 target detection algorithm. The first identification information may be a label used to indicate whether the target object is displayed in the enhanced image. For example, the first identification information may be "the enhanced image displays the target object" or "the enhanced image does not display the target object".
[0059] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem three mentioned in the background technology, "When annotating images of the same target object from different perspectives, the edge of the target object becomes blurred due to changes in the size of the target object and changes in the lighting conditions". Based on this, the present disclosure first segments the first frame image according to the first detection position information to generate a segmented image. Thus, the area corresponding to the first detection position information can be segmented from the first frame image, making the subsequent processing steps more accurate and improving the processing efficiency. Secondly, an image enhancement network is established. Then, the segmented image is input into the input layer of the above image enhancement network to generate a preprocessed image corresponding to the segmented image. Thus, the segmented image can be preprocessed to improve the quality of the segmented image. Next, the preprocessed image is input into the decomposition layer of the above image enhancement network to generate a reflection feature image and an illumination feature image. Thus, the preprocessed image is decomposed into a reflection feature image and an illumination feature image, providing greater flexibility for improving the quality of the image. The quality of the image can be improved for either the reflection feature or the illumination feature. Next, the illumination feature image is input into the denoising layer of the above-mentioned image enhancement network to generate a denoised feature image. Thus, the clarity of the illumination feature image can be improved. Next, the denoised feature image is input into the illumination restoration layer of the above-mentioned image enhancement network to generate an illumination restoration feature image. Thus, the texture of the denoised feature image can be enriched and the local contrast of the denoised feature image can be improved. Next, the illumination restoration feature image and the reflection feature image are input into the output layer of the above-mentioned image enhancement network to generate an enhanced image. Thus, an enhanced image can be obtained. Finally, the enhanced image is identified to generate first identification information characterizing whether the enhanced image shows a target object. In summary, by decomposing the segmented image into an illumination feature image and a reflection feature image, and enhancing the illumination feature image, the quality of the segmented image is improved, the texture of the edge of the target object is clearer, and the details are more prominent, which is conducive to improving the accuracy of image annotation.
[0060] In some optional implementations of some embodiments, the execution subject may extract first object feature information corresponding to the target object in the first frame image based on the first circumscribed rectangular frame through the following steps:
[0061] The first step is to obtain a first edge pixel point set corresponding to the first circumscribed rectangular frame, wherein the first edge pixel point set may be a pixel point set located on the first circumscribed rectangular frame in the first frame image.
[0062] In the second step, for each first edge pixel point in the first edge pixel point set, the following first processing step is performed:
[0063] The first sub-step is to obtain each first area pixel point within the preset range corresponding to the first edge pixel point. In practice, the above-mentioned execution entity may take the area where the circle with the first edge pixel point as the center and the preset radius value as the radius is located as the preset range. Among them, the preset radius value may be a pre-set value. Here, the specific setting of the above-mentioned preset radius value is not limited. Each first area pixel point may be a pixel point located within the preset range, or a pixel point located on the edge of the preset range.
[0064] The second sub-step is to generate a first contrast threshold according to each pixel point in the first area.
[0065] In a third sub-step, in response to a pixel value corresponding to at least one of the first region pixel points being greater than a first comparison threshold, at least one first region pixel point is determined as a first edge feature point set.
[0066] The third step is to determine each determined first edge feature point set as a first feature point set. Each first feature point in the first feature point set corresponds to a first attribute feature. In practice, the execution subject may determine the union of each first edge feature point set as the first feature point set. Each first feature point in the first feature point set may be represented by the coordinates corresponding to the position of the first feature point in the first frame image. The first attribute feature may be a color feature. In practice, for each first feature point, first, the execution subject may determine the neighborhood of the first feature point. The size of the neighborhood may be preset, and the shape of the neighborhood may be a circle or a rectangle. Then, the execution subject may use an image processing library to obtain the color value of each pixel in the neighborhood in the RGB mode to generate a color value set. For example, the image processing library may be PIL (Python Imaging Library). Finally, the execution subject may use PIL to convert the color value set into a color histogram, and determine the color histogram as the color feature corresponding to the first feature point.
[0067] Step 4: for each first feature point in the first feature point set, perform the following second processing step:
[0068] In the first sub-step, the first feature point and the first attribute feature are encoded to generate a first encoded feature point and a first encoded attribute feature. The first attribute feature is the first attribute feature corresponding to the first feature point. The encoding method may be to vectorize the first feature point and the first attribute feature.
[0069] In the second sub-step, the first coding feature point and the first coding attribute feature are spliced to generate a first spliced coding feature. In practice, before splicing the first coding feature point and the first coding attribute feature set, the first coding feature point and the first coding attribute feature set need to be unified on the same vector dimension. The splicing method can be horizontal splicing or vertical splicing.
[0070] The fifth step is to determine each generated first splicing coding feature as the first object feature information.
[0071] In some optional implementations of some embodiments, the execution subject may generate a first contrast threshold according to each pixel point in the first area through the following steps:
[0072] In the first step, an average value of the sum of pixel values corresponding to the pixel points in the first area is determined as the first area threshold.
[0073] In the second step, for each first area pixel point in each first area pixel point, a difference between a pixel value corresponding to the first area pixel point and a first area threshold is determined as a first area difference value, thereby obtaining at least one first area difference value.
[0074] In the third step, for each first area difference value in the at least one first area difference value, the following third processing step is performed:
[0075] The first sub-step is to determine the first distribution probability corresponding to the first region difference. The first distribution probability may be the ratio of the number of occurrences of the first region difference to the number of pixels in each first region. The number of occurrences of the first region difference may be the number of times the difference between the pixel value corresponding to the pixel point in the first region and the first region threshold is the first region difference.
[0076] In the second sub-step, the product of the sum of squares of the first region differences and the first distribution probability is determined as the first sub-variance.
[0077] In the fourth step, the sum of the determined at least one first sub-variance is determined as a first comparison threshold.
[0078] Step 10212, in response to determining that the first frame image is not the initial frame image in the frame image sequence, taking the first frame image as the target proofing image, and continuing to execute the above-mentioned first determination step.
[0079] In some embodiments, in response to determining that the first frame image is not the initial frame image in the frame image sequence, the execution subject may use the first frame image as the target proofing image and continue to execute the first determination step.
[0080] As an example, the target proofing image may be the fifth frame image in the frame image sequence, and the first frame image is the fourth frame image in the frame image sequence. In response to determining that the target proofing image (i.e., the fifth frame image) is not the initial frame image in the frame image sequence, the following second determination step is performed: First, in response to determining that the first frame image (i.e., the fourth frame image) has feature information that matches the object feature information, a first sample frame image is generated according to the fourth frame image. Then, in response to determining that the fourth frame image is not the initial frame image in the frame image sequence, the fourth frame image is used as the target proofing image, and the above-mentioned first determination step is continued. It should be noted that before executing the first determination step again, the object feature information in step 10211 of the first determination step to be executed again needs to be updated. Specifically, the object feature information in step 10211 is updated to the object feature information corresponding to the target object in the fourth frame image. Finally, the first determination step is executed again. Specifically, for the 4th frame image in the frame image sequence, the following first determination step is performed: in response to determining that the 4th frame image is not the initial frame image in the above frame image sequence, the following second determination step is performed: in response to determining that the 3rd frame image has feature information that matches the object feature information, a first sample frame image is generated according to the 3rd frame image. In response to determining that the 3rd frame image is not the initial frame image in the above frame image sequence, the 3rd frame image is used as the target proofing image and the first determination step is continued.
[0081] Step 10213: in response to determining that the first frame image is the initial frame image in the frame image sequence, determining the obtained first sample frame image set as the first candidate sample frame image set.
[0082] In some embodiments, the execution subject may determine the obtained first sample frame image set as the first candidate sample frame image set in response to determining that the first frame image is the initial frame image in the frame image sequence.
[0083] Optionally, before step 10213, the execution entity may further perform the following steps:
[0084] In the first step, in response to the first frame image being the initial frame image in the frame image sequence, it is determined whether the first frame image has feature information matching the feature information of the object.
[0085] The second step is to generate a first sample frame image according to the first frame image in response to the existence of feature information that matches the feature information of the object in response to the existence of the first frame image.
[0086] Step 103: for the target proofing image in the frame image sequence, perform the following third determination step:
[0087] Step 1031, in response to determining that the target proofing image is not the terminating frame image in the frame image sequence, executing the following fourth determination step:
[0088] Step 10311: In response to determining that the second frame image has feature information that matches the object feature information, generate a second sample frame image based on the second frame image.
[0089] In some embodiments, the execution subject may generate a second sample frame image according to the second frame image in response to determining that the second frame image has feature information matching the feature information of the object. The second frame image is a frame image that is adjacent to the corresponding position of the target proofing image in the frame image sequence. The specific implementation of step 10311 is shown in step 10211 and will not be repeated here.
[0090] Step 10312, in response to determining that the second frame image is not the terminating frame image in the frame image sequence, taking the second frame image as the target proofing image, and continuing to execute the third determination step.
[0091] In some embodiments, the execution subject may, in response to determining that the second frame image is not the terminating frame image in the frame image sequence, use the second frame image as the target proofing image and continue to perform the third determination step. The terminating frame image is the last frame image in the frame image sequence.
[0092] As an example, there may be 100 frames of images in the frame image sequence. The target proofing image may be the 5th frame image in the frame image sequence, and the second frame image is the 6th frame image in the frame image sequence. In response to determining that the target proofing image (i.e., the 5th frame image) is not the termination frame image in the frame image sequence, the following fourth determination step is performed: First, in response to determining that the second frame image (i.e., the 6th frame image) has feature information that matches the object feature information, a second sample frame image is generated according to the 6th frame image. Then, in response to determining that the 6th frame image is not the termination frame image in the frame image sequence, the 6th frame image is used as the target proofing image, and the third determination step is continued. It should be noted that before executing the third determination step again, the object feature information in step 10311 of the third determination step to be executed again needs to be updated. Specifically, the object feature information in step 10311 is updated to the object feature information corresponding to the target object in the 6th frame image. Finally, the third determination step is executed again. Specifically, for the sixth frame image in the frame image sequence, the following third determination step is performed: in response to determining that the sixth frame image is not the termination frame image in the above frame image sequence, the following fourth determination step is performed: in response to determining that the sixth frame image has feature information that matches the object feature information, a second sample frame image is generated according to the sixth frame image. In response to determining that the sixth frame image is not the termination frame image in the above frame image sequence, the sixth frame image is used as the target proofing image, and the third determination step is continued.
[0093] Step 10313: in response to determining that the second frame image is the terminal frame image in the frame image sequence, determining the obtained second sample frame image set as the second candidate sample frame image set.
[0094] In some embodiments, the execution subject may determine the obtained second sample frame image set as the second candidate sample frame image set in response to determining that the second frame image is the terminal frame image in the frame image sequence.
[0095] Optionally, before step 10313, the execution entity may further perform the following steps:
[0096] In the first step, in response to the second frame image being the terminal frame image in the frame image sequence, it is determined whether the second frame image has feature information matching the feature information of the object.
[0097] The second step is to generate a second sample frame image according to the second frame image in response to the existence of feature information that matches the feature information of the object in response to the second frame image.
[0098] Step 104: Generate an initial sample set for the target object according to the first candidate sample frame image set and the second candidate sample frame image set.
[0099] In some embodiments, the execution subject may generate an initial sample set for the target object based on the first candidate sample frame image set and the second candidate sample frame image set. In practice, the first candidate sample frame image set and the second candidate sample frame image set may be merged, and the merged set may be used as the initial sample set. In practice, the first candidate sample frame image set and the second candidate sample frame image set may be screened separately as needed and then merged.
[0100] Step 105: Generate a training sample set according to the image similarity between the initial sample in the initial sample set and the target proofing image.
[0101] In some embodiments, the execution subject may generate a training sample set based on the image similarity between the initial sample in the initial sample set and the target proofing image, wherein the image similarity includes semantic similarity and object feature similarity.
[0102] In some optional implementations of some embodiments, the execution subject may generate a training sample set according to the image similarity between the initial sample in the initial sample set and the target proofing image through the following steps:
[0103] The first step is to determine the image semantic information corresponding to the target proofing image as the proofing image semantic information. The image semantic information can be expressed in natural language and mathematical language. For example, the image semantic information can be a vector. In practice, the execution subject can determine the image semantic information corresponding to the target proofing image through a pre-trained neural network. The pre-trained neural network takes an image as input and image semantic information as output. For example, the neural network can be a fully convolutional network.
[0104] In the second step, for each initial sample in the above initial sample set, perform the following generation steps:
[0105] The first sub-step is to determine the image semantic information corresponding to the initial sample as the initial image semantic information. In practice, the execution subject may determine the image semantic information corresponding to the initial sample by using the pre-trained neural network.
[0106] The second sub-step is to determine the object feature information corresponding to the initial sample and for the target object as the sample object feature information. In practice, in response to the initial sample belonging to the first candidate sample frame image set, the execution subject may determine the first object feature information extracted in step 10211 as the sample object feature information. In response to the initial sample belonging to the second candidate sample frame image set, the execution subject may determine the second object feature information extracted in step 10311 as the sample object feature information. Here, the specific implementation method of step 10311 is the same as step 10211, so step 10311 will correspond to the extracted second object feature information.
[0107] The third sub-step is to determine the semantic similarity between the semantic information of the initial image and the semantic information of the proof image. In practice, the execution subject may determine the semantic similarity between the semantic information of the initial image and the semantic information of the proof image by using a cosine similarity method.
[0108] The fourth sub-step is to determine the object feature similarity between the sample object feature information and the object feature information corresponding to the target object. In practice, the execution subject may determine the object feature similarity between the sample object feature information and the object feature information corresponding to the target object by using a cosine similarity method.
[0109] In the third step, at least one initial sample having a semantic similarity less than the first value and an object feature similarity less than the second value is selected from the initial sample set. The first value and the second value may both be pre-set values. The specific settings of the first value and the second value are not limited.
[0110] The fourth step is to determine the sample weight corresponding to each of the at least one initial sample according to the semantic similarity and object feature similarity corresponding to the initial sample. The smaller the semantic similarity and object feature similarity corresponding to the initial sample, the higher the sample weight. In practice, first, the execution entity may determine the product of the semantic similarity and object similarity corresponding to the initial sample as the target similarity. Then, the execution entity may determine the weight corresponding to the target similarity through an inverse proportional function to generate the sample weight.
[0111] Step 5: Generate a training sample set based on the at least one initial sample and the corresponding at least one sample weight, wherein there is a one-to-one correspondence between the at least one initial sample and the at least one sample weight.
[0112] Optionally, after step 105, the execution subject may further perform the following steps:
[0113] In the first step, the training sample set and at least one corresponding sample application scenario information are stored in the target database. The sample application scenario information may be a scenario in which the training sample set is applied. For example, the training sample set may be a frame image set for a certain power bank, and the sample application scenario information may be a scenario in which a power bank is recommended to a target user based on the target user's preferences.
[0114] The second step is to determine the application scenario information corresponding to the target neural network model in response to receiving the model training information for the target neural network model. The target neural network model may be a neural network model to be trained using a training sample set. For example, the neural network model may be a convolutional neural network model. The application scenario information may be a model application scenario corresponding to the target neural network model.
[0115] The third step is, in response to determining that the above-mentioned application scenario information is in the above-mentioned at least one sample application scenario information, performing model training on the above-mentioned target neural network model according to the above-mentioned stored training sample set to generate a trained neural network model.
[0116] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the training sample set generation method of some embodiments of the present disclosure, a certain frame in the video and the video selected by the target user can be used to automatically generate high-quality training sample images for the target object, effectively saving manpower costs and time costs. Specifically, the reason why the annotation results of the training sample images corresponding to the target object are not accurate is that it is often necessary to manually annotate the position, size and outline of the target object in the image, and it is necessary to take into account factors such as the different sizes of the object, lighting conditions, and background changes. It is difficult to accurately and objectively annotate the target object in the image with manual annotation, which leads to low quality of the training sample images corresponding to the target object. Based on this, the training sample set generation method of some embodiments of the present disclosure first obtains a frame image sequence and a target proofing image corresponding to the target video. Among them, the above-mentioned target proofing image is a frame image in the above-mentioned target video that has object feature information corresponding to the target object. Thus, the target video and the target proofing image selected by the target user can be obtained. Then, for the target proofing image in the above frame image sequence, the following first determination step is performed: In response to determining that the target proofing image is not the initial frame image in the above frame image sequence, the following second determination step is performed: First, in response to determining that the first frame image has feature information matching the object feature information, a first sample frame image is generated according to the first frame image. Thus, by matching the feature information corresponding to the first frame image with the object feature information of the target proofing image, it is determined whether the first frame image contains the object feature information corresponding to the target object. If the first frame image has feature information matching the object feature information, a first sample frame image corresponding to the first frame image is generated. Then, in response to determining that the first frame image is not the initial frame image in the above frame image sequence, the first frame image is used as the target proofing image, and the above first determination step is continued. Thus, by cyclically executing the first determination step, a frame image including the object feature information corresponding to the target object before the initially acquired target proofing image can be obtained more accurately, and a corresponding first sample frame image is generated. Finally, in response to determining that the first frame image is the initial frame image in the above frame image sequence, the obtained first sample frame image set is determined as the first candidate sample frame image set. Thus, all first sample frame images generated by the first determination step can be obtained. Secondly, for the target proofing image in the above frame image sequence, the following third determination step is performed: In response to determining that the target proofing image is not the termination frame image in the above frame image sequence, the following fourth determination step is performed: First, in response to determining that the second frame image has feature information that matches the object feature information, a second sample frame image is generated based on the second frame image. Thus, by matching the feature information corresponding to the second frame image with the target proofing image object feature information, it is determined whether the second frame image contains the object feature information corresponding to the target object.If the second frame image has feature information that matches the object feature information, a first sample frame image corresponding to the second frame image is generated. Then, in response to determining that the second frame image is not the termination frame image in the above frame image sequence, the second frame image is used as the target proofing image, and the above third determination step is continued. Thus, by cyclically executing the third determination step, a frame image including the object feature information corresponding to the target object after the initially acquired target proofing image can be obtained more accurately, and a corresponding second sample frame image is generated. Finally, in response to determining that the second frame image is the termination frame image in the above frame image sequence, the obtained second sample frame image set is determined as the second candidate sample frame image set. Thus, all second sample frame images generated by the third determination step can be obtained. Then, based on the above first candidate sample frame image set and the above second candidate sample frame image set, an initial sample set for the target object is generated. Thus, all frame images in the target video that have the object feature information corresponding to the target object can be obtained. Then, based on the image similarity between the initial sample and the target proofing image in the above initial sample set, a training sample set is generated. Thus, the frame images in the initial sample set that are not much different from the target proofing image can be removed, which reduces the redundant images in the training sample set, improves the quality of the training sample set, and thus reduces the storage cost. In summary, by continuously updating the target proofing image and matching the feature information of the first frame image or the second frame image with the object feature information, it can be more accurately determined whether the first frame image or the second frame image can become a candidate sample frame image. And by removing the redundant images in the training sample set through the image similarity between the initial sample in the initial sample set and the target proofing image, the quality of the training sample set is further improved, effectively saving labor costs and time costs.
[0117] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a training sample set generation device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the training sample set generating device can be specifically applied to various electronic devices.
[0118] like Figure 2As shown, a training sample set generation device 200 includes: an acquisition unit 201, a first determination unit 202, a second determination unit 203, a first generation unit 204, and a second generation unit 205. The acquisition unit 201 is configured to: acquire a frame image sequence and a target proofing image corresponding to a target video, wherein the target proofing image is a frame image in the target video that has object feature information corresponding to a target object. The first determination unit 202 is configured to: for the target proofing image in the frame image sequence, perform the following first determination step: in response to determining that the target proofing image is not an initial frame image in the frame image sequence, perform the following second determination step: in response to determining that the first frame image has feature information matching the object feature information, generate a first sample frame image according to the first frame image; in response to determining that the first frame image is not an initial frame image in the frame image sequence, use the first frame image as a target proofing image, and continue to perform the above first determination step; in response to determining that the first frame image is an initial frame image in the frame image sequence, determine the obtained first sample frame image set as a first candidate sample frame image set. The second determination unit 203 is configured to: for the target proofing image in the above frame image sequence, perform the following third determination step: in response to determining that the target proofing image is not the termination frame image in the above frame image sequence, perform the following fourth determination step: in response to determining that the second frame image has feature information matching the feature information of the object, generate a second sample frame image according to the second frame image; in response to determining that the second frame image is not the termination frame image in the above frame image sequence, take the second frame image as the target proofing image and continue to perform the above third determination step; in response to determining that the second frame image is the termination frame image in the above frame image sequence, determine the obtained second sample frame image set as the second candidate sample frame image set. The first generation unit 204 is configured to: generate an initial sample set for the target object according to the first candidate sample frame image set and the second candidate sample frame image set. The second generation unit 205 is configured to: generate a training sample set according to the image similarity between the initial sample in the above initial sample set and the above target proofing image.
[0119] It can be understood that the units recorded in the training sample set generation device 200 and the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the training sample set generation device 200 and the units contained therein, and will not be described in detail here.
[0120] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0121] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0122] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0123] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.
[0124] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0125] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0126] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a frame image sequence and a target proofing image corresponding to the target video, wherein the target proofing image is a frame image in the target video that contains object feature information corresponding to the target object; for the target proofing image in the frame image sequence, performs the following first determination step: in response to determining that the target proofing image is not the initial frame image in the frame image sequence, performs the following second determination step: in response to determining that the first frame image contains feature information that matches the object feature information, generates a first sample frame image based on the first frame image; in response to determining that the first frame image is not the initial frame image in the frame image sequence, uses the first frame image as the target proofing image and continues to perform the first determination step; in response to determining that the first frame image is the initial frame image in the frame image sequence, uses the obtained first sample frame image as the target proofing image. The set is determined as the first candidate sample frame image set; for the target proofing image in the above frame image sequence, the following third determination step is performed: in response to determining that the target proofing image is not the terminal frame image in the above frame image sequence, the following fourth determination step is performed: in response to determining that the second frame image has feature information matching the feature information of the object, a second sample frame image is generated according to the second frame image; in response to determining that the second frame image is not the terminal frame image in the above frame image sequence, the second frame image is used as the target proofing image, and the above third determination step is continued; in response to determining that the second frame image is the terminal frame image in the above frame image sequence, the obtained second sample frame image set is determined as the second candidate sample frame image set; based on the above first candidate sample frame image set and the above second candidate sample frame image set, an initial sample set for the target object is generated; based on the image similarity between the initial sample in the above initial sample set and the above target proofing image, a training sample set is generated.
[0127] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0129] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a first determination unit, a second determination unit, a first generation unit, and a second generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring a frame image sequence and a target proofing image corresponding to a target video".
[0130] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0131] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.
Claims
1. A method for generating a training sample set, comprising: Acquire a frame image sequence and a target proofing image corresponding to a target video, wherein the target proofing image is a frame image in the target video that contains object feature information corresponding to a target object; For the target proofing image in the frame image sequence, the following first determination step is performed: In response to determining that the target proofing image is not an initial frame image in the frame image sequence, performing the following second determination step: In response to determining that the first frame image has feature information matching the feature information of the object, generating a first sample frame image according to the first frame image; In response to determining that the first frame image is not an initial frame image in the frame image sequence, taking the first frame image as a target proofing image and continuing to perform the first determining step; In response to determining that the first frame image is an initial frame image in the frame image sequence, determining the obtained first sample frame image set as a first candidate sample frame image set; For the target proofing image in the frame image sequence, the following third determination step is performed: In response to determining that the target proofing image is not the terminating frame image in the frame image sequence, performing the following fourth determination step: In response to determining that the second frame image has feature information matching the feature information of the object, generating a second sample frame image according to the second frame image; In response to determining that the second frame image is not the terminating frame image in the frame image sequence, taking the second frame image as the target proofing image and continuing to perform the third determining step; In response to determining that the second frame image is the terminal frame image in the frame image sequence, determining the obtained second sample frame image set as a second candidate sample frame image set; generating an initial sample set for the target object according to the first candidate sample frame image set and the second candidate sample frame image set; A training sample set is generated according to the image similarity between the initial samples in the initial sample set and the target proofing image.
2. The method according to claim 1, wherein: The first frame image is a frame image that is located before and adjacent to the corresponding position of the target proofing image in the frame image sequence, and; In response to determining that the first frame image has feature information matching the feature information of the object, generating a first sample frame image according to the first frame image includes: According to the position information corresponding to the target object in the target proofing image, the position information corresponding to the target object in the first frame image is detected to generate first detected position information; Identify the area corresponding to the first detection position information to generate first identification information; In response to determining that the first identification information indicates that the area corresponding to the first detection position information displays the target object, segmenting the area corresponding to the first detection position information to generate a first circumscribed rectangular frame corresponding to the target object in the first frame image; Extracting first object feature information corresponding to the target object in the first frame image based on the first circumscribed rectangular frame; A first sample frame image is generated according to the first object feature information and the first circumscribed rectangular frame.
3. The method according to claim 2, wherein: The step of extracting first object feature information corresponding to the target object in the first frame image based on the first circumscribed rectangular frame includes: Obtain a first edge pixel point set corresponding to a first circumscribed rectangular frame; For each first edge pixel point in the first edge pixel point set, the following first processing step is performed: Acquire each first region pixel point within a preset range corresponding to the first edge pixel point; Generate a first contrast threshold according to each pixel point in the first area; In response to a pixel value corresponding to at least one of the first region pixel points being greater than a first comparison threshold, determining at least one first region pixel point as a first edge feature point set; Determine each determined first edge feature point set as a first feature point set, wherein each first feature point in the first feature point set corresponds to a first attribute feature; For each first feature point in the first feature point set, the following second processing step is performed: Encoding the first feature point and the first attribute feature to generate a first encoded feature point and a first encoded attribute feature; splicing the first coding feature point and the first coding attribute feature to generate a first spliced coding feature; The generated first splicing coding features are determined as first object feature information.
4. The method according to claim 3, wherein: The step of generating a first contrast threshold according to each pixel point in the first area includes: Determine an average value of the sum of pixel values corresponding to each pixel point in the first area as a first area threshold; For each first-region pixel point in each first-region pixel point, determine the difference between the pixel value corresponding to the first-region pixel point and the first-region threshold as the first-region difference value, and obtain at least one first-region difference value; For each first area difference value of the at least one first area difference value, the following third processing step is performed: Determine a first distribution probability corresponding to the first area difference; Determine the product of the sum of squares of the first region differences and the first distribution probability as a first sub-variance; The sum of the determined at least one first subvariance is determined as a first comparison threshold.
5. The method according to claim 1, wherein: Before determining the obtained first sample frame image set as the first candidate sample frame image set in response to determining that the first frame image is the initial frame image in the frame image sequence, the method further includes: In response to the first frame image being an initial frame image in the frame image sequence, determining whether the first frame image has feature information matching the feature information of the object; In response to the existence, a first sample frame image is generated according to the first frame image.
6. The method according to claim 1, wherein: The image similarity includes semantic similarity and object feature similarity, and; The generating of the training sample set according to the image similarity between the initial sample in the initial sample set and the target proofing image comprises: Determining image semantic information corresponding to the target proofing image as proofing image semantic information; For each initial sample in the initial sample set, the following generation steps are performed: Determining image semantic information corresponding to the initial sample as initial image semantic information; Determine object feature information corresponding to the initial sample and for the target object as sample object feature information; Determining the semantic similarity between the initial image semantic information and the proofing image semantic information; Determining the object feature similarity between the sample object feature information and the object feature information corresponding to the target object; Filtering at least one initial sample from the initial sample set, the initial sample having a semantic similarity less than a first value and an object feature similarity less than a second value; For each initial sample of the at least one initial sample, determining a sample weight corresponding to the initial sample according to a semantic similarity and an object feature similarity corresponding to the initial sample; The training sample set is generated according to the at least one initial sample and the corresponding at least one sample weight.
7. The method according to claim 1, wherein: The method further comprises: Correspondingly storing the training sample set and at least one sample application scenario information in a target database; In response to receiving model training information for a target neural network model, determining application scenario information corresponding to the target neural network model; In response to determining that the application scenario information is within the at least one sample application scenario information, model training is performed on the target neural network model according to the stored training sample set to generate a trained neural network model.
8. A training sample set generating device, comprising: An acquisition unit is configured to acquire a frame image sequence and a target proofing image corresponding to a target video, wherein the target proofing image is a frame image in the target video having object feature information corresponding to a target object; A first determining unit is configured to perform the following first determining step for a target proofing image in the frame image sequence: in response to determining that the target proofing image is not an initial frame image in the frame image sequence, perform the following second determining step: in response to determining that the first frame image has feature information matching the feature information of the object, generate a first sample frame image according to the first frame image; in response to determining that the first frame image is not an initial frame image in the frame image sequence, use the first frame image as the target proofing image and continue to perform the first determining step; in response to determining that the first frame image is an initial frame image in the frame image sequence, determine the obtained first sample frame image set as a first candidate sample frame image set; The second determining unit is configured to perform the following third determining step for a target proofing image in a frame image sequence: in response to determining that the target proofing image is not a termination frame image in the frame image sequence, perform the following fourth determining step: in response to determining that the second frame image has feature information matching the feature information of the object, generate a second sample frame image according to the second frame image; in response to determining that the second frame image is not a termination frame image in the frame image sequence, use the second frame image as the target proofing image and continue to perform the third determining step; in response to determining that the second frame image is a termination frame image in the frame image sequence, determine the obtained second sample frame image set as a second candidate sample frame image set; A first generating unit is configured to generate an initial sample set for the target object according to the first candidate sample frame image set and the second candidate sample frame image set; The second generating unit is configured to generate a training sample set according to the image similarity between the initial sample in the initial sample set and the target proofing image.
9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
