Image set generation method, device, equipment and computer readable storage medium

Abstract

Embodiments of the present disclosure provide a method and device for generating an image set, an apparatus and a computer readable storage medium. The image set is used for training a detection model for an abnormal organism in a restricted area, and the image set includes a plurality of sample images. The method for generating the image set includes generating each sample image according to the following steps: obtaining at least one target image from a pre-obtained first image set, the target image being an image of an abnormal organism segmented from a first image in the first image set; and synthesizing the target image with a background image to obtain a sample image, wherein the background image is obtained by photographing the restricted area.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and in particular, to a method and apparatus for generating an image set, and a computer readable storage medium. BACKGROUND

[0002] For the sake of life or property safety, some areas are strictly prohibited for pedestrians or animals to appear in order to avoid forming a safety hazard. Therefore, it is necessary to detect abnormal organisms in the forbidden area where personnel or animals are prohibited to appear.

[0003] In related technologies, a target detection algorithm based on a deep learning network is used to detect whether abnormal organisms appear in the forbidden area. The deep learning network usually needs a large amount of data set to train and generate. For the forbidden area, most of the images taken are not abnormal organisms. Therefore, even if the above target detection algorithm has high accuracy, the detection accuracy of the trained deep learning network cannot meet the detection requirements when a large number of images without abnormal targets (abnormal organisms) are used to train the deep learning network. SUMMARY

[0004] The present disclosure provides a method and apparatus for generating an image set, and a computer readable storage medium.

[0005] In a first aspect, the present disclosure provides a method for generating an image set, the image set being used to train a detection model for abnormal organisms in a forbidden area, the image set including a plurality of sample images; the method comprising generating each sample image according to the following steps:

[0006] According to a first image set obtained in advance, at least one target image is obtained, the target image being an image of the abnormal organism segmented from a first image in the first image set, the first image set including a plurality of first images;

[0007] The target image is synthesized with a background image to obtain the sample image, wherein the background image is obtained by photographing the forbidden area.

[0008] In some embodiments, according to a first image set obtained in advance, at least one target image is obtained, comprising:

[0009] Each first image in the first image set is subjected to instance segmentation processing to obtain a target image set corresponding to each first image; the target image set including at least one target image.

[0010] In some embodiments, the instance segmentation processing of each first image in the first image set to obtain a target image set corresponding to each first image comprises:

[0011] inputting the first image into a Mask R-CNN instance segmentation network for processing to obtain the target image set.

[0012] In some embodiments, the background image is obtained by photographing the forbidden area using a target device,

[0013] The synthesizing the target image and the background image to obtain the sample image comprises:

[0014] According to the corresponding coordinates of the position to be pasted, the device parameters of the target device, the image size of the background image, and the preset size of the abnormal organism, a target size parameter for synthesizing the target image into the background image is determined; the position to be pasted is the position of the target image in the background image when the target image is synthesized with the background image;

[0015] According to the target size parameter, the target image is adjusted in size to obtain an adjusted target image.

[0016] The adjusted target image and the background image are synthesized to obtain the sample image.

[0017] In some embodiments, before the target size parameter is determined, the method further comprises:

[0018] The background image is subjected to semantic segmentation processing to determine a plurality of image regions in the background image; one of the plurality of image regions is determined as a target region.

[0019] Any position in the target region is determined as the position to be pasted.

[0020] In some embodiments, the semantic segmentation processing of the background image to determine a plurality of image regions in the background image comprises:

[0021] The background image is input into a U-Net semantic segmentation network for processing to obtain a plurality of image regions.

[0022] In some embodiments, the device parameters of the target device at least include: the installation height of the target device, the focal length of the target device, the angle between the optical axis of the target device and the vertical direction,

[0023] The determination of the target size parameter for synthesizing the target image into the background image according to the device parameters of the target device, the image size of the background image, and the preset size of the abnormal organism comprises:

[0024] determine a first angle according to the corresponding coordinates of the position to be pasted and a focal length of the target device, the first angle being an included angle between a line connecting a position where the target device is located and a bottom position of the abnormal organism and an optical axis of the target device;

[0025] determine a second angle according to the first angle, an installation height of the target device, an included angle between the optical axis of the target device and a vertical direction, and a preset size of the abnormal organism, the preset size of the abnormal organism being determined according to a type of the abnormal organism, the second angle being an included angle between a first line and a second line, the first line being the line connecting the position where the target device is located and the bottom position of the abnormal organism, and the second line being a line connecting the position where the target device is located and a top position of the abnormal organism;

[0026] determine the target size parameter according to the first angle, the second angle, and the image size.

[0027] In some embodiments, the size adjustment of the target image according to the size parameter to obtain an adjusted target image comprises:

[0028] determine an adjustment ratio of the target image according to the image size and the target size parameter;

[0029] respectively adjust a width and a height of the target image according to the adjustment ratio to obtain the adjusted target image.

[0030] In some embodiments, the image synthesis of the adjusted target image and the background image to obtain the sample image comprises:

[0031] determine a calibration position in the adjusted target image according to the type of the abnormal organism;

[0032] paste the adjusted target image into the background image to obtain a first image, in the first image, the calibration position is aligned with a position to be pasted of the background image;

[0033] perform color adjustment on the first image to obtain the sample image, the color adjustment comprising brightness adjustment and / or chroma adjustment.

[0034] In some embodiments, the color adjustment on the first image to obtain the sample image comprises:

[0035] set an area in the first image, which is located outside a region where the target image is located, to a first preset color to obtain a second image;

[0036] input the first image and the second image into a color neural network to perform color adjustment on the first image to obtain the sample image.

[0037] In a second aspect, the embodiments of the present disclosure provide a device for generating an image set, the image set being used for training a detection model for an abnormal organism in a forbidden area, the image set comprising a plurality of sample images; the device comprises:

[0038] An acquisition module is configured to acquire at least one target image according to a first image set acquired in advance, the target image being an image of the abnormal organism segmented from a first image in the first image set, the first image set comprising a plurality of first images;

[0039] A processing module is configured to synthesize the target image and a background image to obtain the sample image, wherein the background image is obtained by photographing the forbidden area.

[0040] In a third aspect, the embodiments of the present disclosure provide a device for generating an image set, comprising a memory and a processor, the memory storing a computer program, and the computer program is executed by the processor to implement the method of the first aspect.

[0041] In a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, and are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation on the present disclosure. In the drawings:

[0043] Figure 1 A flowchart of a method for generating an image set provided by the embodiments of the present disclosure.

[0044] Figure 2 A flowchart of another method for generating an image set provided by the embodiments of the present disclosure.

[0045] Figure 3 A framework diagram of a Mask R-CNN instance segmentation network provided by the embodiments of the present disclosure.

[0046] Figure 4a An original image provided by the embodiments of the present disclosure.

[0047] Figure 4b An image after instance segmentation processing on the Figure 4a

[0048] ​Figure 5 A schematic diagram of a target image set provided by an embodiment of the present disclosure.

[0049] Figure 6 A schematic diagram of a framework of a U-Net semantic segmentation network provided by an embodiment of the present disclosure.

[0050] Figure 7 An image after semantic segmentation processing on Figure 4a

[0051] Figure 8 A schematic diagram of a camera imaging principle provided by an embodiment of the present disclosure.

[0052] Figure 9 A flowchart of a generation method of another image set provided by an embodiment of the present disclosure.

[0053] Figure 10 A schematic diagram of a framework of a RainNet neural network provided by an embodiment of the present disclosure.

[0054] Figure 11 A structural schematic diagram of a generation apparatus of an image set provided by an embodiment of the present disclosure.

[0055] Figure 12 A structural schematic diagram of a generation device of an image set provided by an embodiment of the present disclosure.

[0056] Figure 13 A structural schematic diagram of a computer readable storage medium provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the present disclosure, and are not intended to limit the present disclosure.

[0058] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without any inventive effort fall within the scope of protection of the present disclosure.

[0059] ​Unless otherwise defined, technical terms and scientific terms used in the present disclosure have the meanings commonly understood by one of ordinary skill in the art to which the present disclosure pertains. The terms "first", "second", and similar terms do not imply any order, quantity, or importance, but are used to distinguish different components. Also, the terms "include", "comprise", and similar terms are intended to mean that the elements or objects listed after the terms encompass the elements or objects recited after the terms, and equivalents thereof, without precluding other elements or objects. The terms "connected" or "coupled" and similar terms are not limited to physical or mechanical connections or couplings, but also include electrical connections or couplings, whether direct or indirect. The terms "upper", "lower", "left", "right", and similar terms are used only to indicate relative positions, and when the absolute positions of the described objects are changed, the relative positions can also be changed accordingly.

[0060] For the consideration of life or property safety, some areas are strictly prohibited for pedestrians or animals to appear in order to avoid forming safety hazards, and therefore, it is necessary to detect abnormal organisms in the forbidden area where personnel or animals are prohibited to appear.

[0061] In the related art, a target detection algorithm based on a deep learning network is used to detect whether abnormal organisms appear in the forbidden area. The deep learning network usually needs a large amount of data set to train and generate, and for the forbidden area, most of the images taken are not abnormal organisms, so even if the above target detection algorithm has high accuracy, the detection accuracy of the trained deep learning network obtained by training the deep learning network using a large number of images without abnormal organisms cannot meet the detection requirements.

[0062] Therefore, in the related art, it is proposed that an image set can be generated by image synthesis, specifically, a target image including an abnormal organism and a background image including a forbidden area environment are synthesized to generate an image set to train a deep learning network, and improve the detection accuracy of the deep learning network. However, the above image synthesis method has the problems of target image proportion being inconsistent, resulting in the target image being too conspicuous, and the chroma and / or brightness of the target image and the background image being inconsistent, resulting in the image being unrealistic.

[0063] In order to solve at least one of the above technical problems, the present disclosure provides a method for generating an image set for training a detection model for abnormal organisms in a forbidden area.

[0064] Figure 1 A flowchart of a method for generating an image set provided by the present disclosure is shown in FIG. 1. The image set includes a plurality of sample images. As shown in FIG. 1, the method for generating an image set includes generating each sample image according to the following steps: Figure 1 ​

[0065] S1, obtaining at least one target image from a first image set, the target image being an image of an abnormal organism segmented from a first image in the first image set, the first image set including a plurality of first images.

[0066] It should be noted that the first image set is selected from the existing image set, and each first image in the first image set is an image with an abnormal organism. In addition, the abnormal organism herein refers to a biological organism prohibited from entering the forbidden area, such as a human or other animals.

[0067] S2, synthesizing the target image with a background image to obtain a sample image, wherein the background image is obtained by photographing the forbidden area.

[0068] The image set generation method provided by the embodiments of the present disclosure obtains a target image from a first image set, and synthesizes the target image with an abnormal organism with a background image with a forbidden area environment to obtain a sample image. Since the sample image contains an abnormal organism, the detection model trained using a plurality of sample images can improve the detection accuracy when detecting abnormal organisms in the forbidden area.

[0069] Figure 2 Another flowchart of the image set generation method provided by the embodiments of the present disclosure is shown in some embodiments, as shown in Figure 2 The step S1 can include:

[0070] S11, performing instance segmentation processing on each first image in the first image set to obtain a target image set corresponding to each first image; the target image set includes at least one target image. The instance segmentation refers to that different instances are framed in the image according to the target detection method, and each pixel in the different instances is labeled by a semantic segmentation algorithm to segment at least one target.

[0071] Since the accuracy of instance segmentation has certain requirements in the generation process of the target image, and the speed of instance segmentation is not limited, the two-stage network with high segmentation accuracy is used for instance segmentation processing in the embodiments of the present disclosure, such as Mask R-CNN instance segmentation network.

[0072] Optionally, the step S11 can include: inputting the first image into the pre-trained Mask R-CNN instance segmentation network for processing to obtain the target image set. Since the first image can include at least one abnormal organism, at least one target image can be segmented after the instance segmentation processing, and the target image set is formed.

[0073] Figure 3A schematic diagram of the framework of a Mask R-CNN instance segmentation network provided in this disclosure embodiment is shown below. Figure 3 As shown, the image is input into the Region of Interest (ROI Align) network, where it undergoes pooling using the bilinear interpolation algorithm to obtain a feature map. Based on the size of the ROI and the degree of pooling, the feature map is divided into multiple candidate boxes. Finally, a convolution operation is performed to achieve accurate segmentation of the input image. The advantage of the Mask R-CNN instance segmentation network lies in its use of the ROI Align operation, i.e., the bilinear interpolation algorithm, which eliminates quantization errors, ensuring complete alignment between pixels in the input image and pixels in the feature image, thus improving detection accuracy.

[0074] Figure 4a The original image provided for the embodiments of this disclosure. Figure 4b To Figure 4a The image after instance segmentation. In one example, such as... Figure 4a , Figure 4b As shown, based on the Mask R-CNN instance segmentation network, instance segmentation technology can be used to segment the target people in the image and distinguish different target people.

[0075] Figure 5 This is a schematic diagram of the target image set provided in an embodiment of this disclosure. Each target image in the target image set can be stored in PNG format with a transparent background. It should be noted that... Figure 5 The multiple target images in the target image set shown can be segmented from a first image or segmented from multiple first images. This disclosure does not limit this.

[0076] In some embodiments, such as Figure 2 As shown, before step S21, the method for synthesizing the image set may further include:

[0077] S20, perform semantic segmentation on the background image to determine multiple image regions in the background image; take one of the multiple image regions as the target region; and determine any position in the target region as the pasting position.

[0078] The process of combining the target image and the background image can be simply viewed as pasting the target image onto the background image; the pasting position refers to the placement of the target image on the background image during the combination process. For scenarios involving the detection of abnormal biological intrusions in restricted areas, it is necessary to appropriately select the placement position of the target image on the background image, i.e., the pasting position, to ensure the authenticity of the sample image.

[0079] Specifically, semantic segmentation technology is used to divide different regions in the background image, and the region to be pasted is selected based on the segmentation results. For example, in a lake water spill detection scenario, the target image is pasted only in the lake area; in a lawn trampling detection scenario, the target image is pasted only in the lawn area.

[0080] In this embodiment, the U-Net semantic segmentation network is used to process the background image to ensure the accuracy of the segmented image regions. In some embodiments, S20 performs semantic segmentation processing on the background image to determine multiple image regions in the background image. Specifically, this may include: inputting the background image into a pre-trained U-Net semantic segmentation network for processing to obtain multiple image regions.

[0081] Figure 6 This is a schematic diagram of the framework of a U-Net semantic segmentation network provided in an embodiment of this disclosure, as shown below. Figure 6 As shown, the U-Net semantic segmentation network includes a first module 1, a second module 2, and a third module 3. The first module 1 includes multiple first units 11, each first unit 11 including multiple first networks 11a and a pooling network 11b. The second module includes multiple second units 12, each second unit 12 including an upsampling network 12a and multiple first networks 11a, and each second unit 12 has a corresponding first unit 11. The third module 3 includes a regression network 31 (Softmax). The aforementioned first network 11a, through convolution and batch normalization operations combined with the ReLU activation function, transforms a low-resolution image containing high-dimensional features into a high-resolution image while preserving the high-dimensional features.

[0082] Specifically, in the first module 1, the input of the first unit 11 is the original image, and the inputs of the other units 11 are the output images of the previous unit 11. After feature extraction through continuous convolution and pooling, each unit 11 inputs its feature image to the corresponding second unit 12. In the second module 2, the inputs of the second units 12 (excluding the first one) also include the feature images processed by the previous unit 12. That is, the second units 12 (excluding the first one) fuse the feature images input from the first unit 11 and the previous unit 12, then perform upsampling using an activation function, and finally input the processed feature image to the third module 3 via the last unit 12. The regression network 31 in the third module 3 calculates the loss function, and if the loss function meets the preset requirements, the final region segmentation result is output.

[0083] Figure 7 To Figure 4aThe image after semantic segmentation processing. In one example, as shown in Figure 4a , Figure 7 , based on the U-Net semantic segmentation network, the scene in Figure 4a is divided into different regions, and the segmentation result is as shown in Figure 7 , wherein the regions where the crowd, trees, grass, and sky are located in the image can be represented by different colors.

[0084] In some embodiments, as shown in Figure 2 , the above step S2 can include steps S21-S23, wherein:

[0085] S21, according to the corresponding coordinates of the position to be pasted, the device parameters of the target device, the image size of the background image, and the preset size of the abnormal organism, determine the target size parameter of the target image synthesized into the background image; the position to be pasted is the position of the target image in the background image when the target image is synthesized into the background image. The corresponding coordinates of the position to be pasted refer to the coordinates of the position to be pasted in the background image.

[0086] Wherein, the target device is a device for shooting the forbidden area to obtain the background image.

[0087] The determination of the above target size parameter is based on the following considerations: first, under the condition that the target device and the shooting angle are determined, the object being shot is located at different positions in the shooting environment, which will form different proportions in the imaging picture relative to the background environment. For example, in the same background, the target body is closer to the lens, and the picture proportion occupied in the imaging picture is higher. Secondly, the device parameters of different devices are different, so even if the same scene is shot at the same position, the picture effect formed will be different. Finally, the target size parameter is the size of the target image presented in the background image, so in order to ensure the authenticity of the image, the preset size of the abnormal organism and the image size of the background image also need to be considered to maximize the approximation to the real acquisition image.

[0088] The above target size parameter can be a height parameter of the target image synthesized into the background image, or a width parameter of the target image synthesized into the background image, which is not limited in the embodiments of the present disclosure.

[0089] The determination process of the target size parameter will be described below with the target size parameter as an example of the height parameter.

[0090] Figure 8 The schematic diagram of the camera imaging principle provided by the embodiments of the present disclosure, combined with the above analysis, as shown in Figure 8As shown, the corresponding coordinates O' of the to-be-pasted position, the device parameters of the target device, the image size image_h of the background image, and the preset size h of the abnormal organism all have an impact on the target size parameter. The device parameters of the target device at least include the installation height H of the target device, the focal length f of the target device, and the included angle θ between the optical axis of the target device and the vertical direction.

[0091] It should be noted that, as Figure 8 As shown, based on the camera imaging principle, when the position of the abnormal organism is CD, the position of the abnormal organism in the image after being photographed by the target device arranged at the O point is O'A.

[0092] In some embodiments, step S21 can specifically include:

[0093] According to the corresponding coordinates of the to-be-pasted position and the focal length of the target device, a first angle a is determined, which can be specifically represented by formula 1. The first angle a refers to the included angle between the line OC connecting the position O of the target device and the bottom position C of the abnormal organism and the optical axis of the target device.

[0094]

[0095] According to the first angle a, the installation height H of the target device, the included angle θ between the optical axis of the target device and the vertical direction, and the preset size h of the abnormal organism, a second angle β is determined, which can be specifically represented by formula 2. The second angle β refers to the included angle between the first line and the second line, the first line being the line OC connecting the position O of the target device and the bottom position of the abnormal organism, and the second line being the line OD connecting the position O and the top position D of the abnormal organism.

[0096] In addition, the preset size of the abnormal organism refers to a size (here, the size can specifically refer to the height) related to the type of the abnormal organism and close to the actual size of the abnormal organism. The preset size of the abnormal organism is determined according to the type of the abnormal organism, and the preset size of the same type of abnormal organism is the same. For example, according to a pre-stored mapping relationship table of the preset size and the type of the organism, the preset size of the abnormal organism is determined. The mapping relationship table can store preset sizes corresponding to various types of organisms, for example, when the abnormal organism is a pedestrian, the preset size can be set to 1.6 m or 1.75 m, etc.; when the abnormal organism is a dog, the preset size can be set to 0.3 m or 0.5 m, etc.

[0097]

[0098] According to the first angle a, the second angle β, and the image size image_h, a target size parameter AB is determined, which can be specifically represented by formula 3.

[0099] It should be noted that when the target size parameter is a height parameter, the image size is the height of the image, and when the target size parameter is a width parameter, the image size is the width of the image.

[0100]

[0101] S22, size adjustment is performed on the target image according to the target size parameter, to obtain an adjusted target image.

[0102] In some embodiments, step S22 can specifically include:

[0103] According to the image size and the target size parameter, an adjustment ratio of the target image is determined; and according to the adjustment ratio, width adjustment and height adjustment are respectively performed on the target image, to obtain the adjusted target image.

[0104] In one example, the image size of the target image is a x b, and the target size parameter is c. In the case where the target size parameter is a height parameter, the adjustment ratio is c / a, and according to the adjustment ratio, the height of the target image is adjusted to c, and the width is adjusted to c / a x b, to obtain the adjusted target image. In the case where the target size parameter is a width parameter, the adjustment ratio is c / b, and according to the adjustment ratio, the width of the target image is adjusted to c, and the height is adjusted to c / b x a, to obtain the adjusted target image.

[0105] S23, image synthesis is performed on the adjusted target image and the background image, to obtain a sample image.

[0106] The method for generating an image set provided by the embodiments of the present disclosure includes obtaining a target image set containing at least one target image through instance segmentation in a first image set obtained in advance. Since the background image is obtained by capturing the forbidden area by a camera device, based on the imaging principle, a target size parameter of the target image on the background image is determined, and the image size of the target image is adjusted according to the target size parameter, to improve the authenticity of the sample image after the target image and the background image are synthesized.

[0107] Figure 9 The flowchart of another method for generating an image set provided by the embodiments of the present disclosure is shown in some embodiments, as shown in Figure 9 Step S23 can specifically include steps S231-S233.

[0108] S231, according to the type of the abnormal organism, a calibration position in the adjusted target image is determined.

[0109] In one example, the forbidden area prohibits personnel from passing through, and the type of the abnormal organism is human. Therefore, the calibration position in the adjusted target image is the position of the feet of the person in the image.

[0110] S232, paste the adjusted target image into the background image to obtain a first image, in which the calibration position is aligned with the to-be-pasted position of the background image.

[0111] S233, perform color adjustment on the first image to obtain a sample image, and the color adjustment includes brightness adjustment and / or chroma adjustment.

[0112] After the adjusted target image is pasted onto the background image, due to different lightings and the like, the color of the first image formed after pasting is not coordinated, and therefore, the color of the first image is adjusted by taking the lighting of the background image as a reference and using a color neural network.

[0113] In some embodiments, step S233 can specifically include:

[0114] The area in the first image located outside the area where the target image is located is set to a first preset color to obtain a second image. The first image and the second image are input into the color neural network to perform color adjustment on the first image to obtain a sample image. In one example, the first preset color is black, that is, the pixels of the area in the first image located outside the area where the target image is located are set to 0.

[0115] Specifically, the color neural network can use a RainNet neural network to perform style transfer on the pasted target image based on the background image, so that the target image is more integrated with the background image. Figure 10 A framework schematic diagram of a RainNet neural network provided by the embodiments of the present disclosure is shown in Figure 10 The RainNet neural network includes a first convolution module 4, a second convolution module 5, a third convolution module 6, and a deconvolution module 7. The first convolution module 4 includes one convolution network 41. The deconvolution module 7 includes one deconvolution network 71. The second convolution module 5 includes multiple second convolution networks 51 based on a modified activation function (LReLU). The third convolution module 6 includes multiple cascaded convolution units, specifically including multiple first convolution units 61, multiple second convolution units 62, and one third convolution unit 63. The first convolution unit 61 includes one second convolution network 51 and one deconvolution network 61a based on an activation function. The second convolution unit 62 includes one second convolution network 51, one deconvolution network 61a based on an activation function, and one attention self-made network 62a. The third convolution unit 63 includes one deconvolution network 61a based on an activation function, one convolution network 41, and one attention self-made network 62a.

[0116] Specifically, the first image Ic is subjected to multi-layer convolution processing by the first convolution module 4 and the second convolution module 5 in the RainNet neural network, high-dimensional features are extracted and input to the third convolution module 6, the third convolution module 6 takes the first image input by each level of the second convolution network 51 and the second image M of the same resolution as the input, Ic x (1-M) obtains the background region in Ic, Ic x M obtains the foreground region in Ic, and after the deconvolution module 7, the statistical style parameters γ i and β i are obtained i and β i are multiplied, and are added to the normalized foreground features in a channel manner to achieve color balance, so that the picture content in the sample image is more coordinated, and the authenticity of the sample image is improved.

[0117] Figure 11 A structure diagram of an image set generation device provided by an embodiment of the present disclosure is shown in the figure. As shown in the figure, the image set generation device comprises an acquisition module 10 and a processing module 20. Figure 11

[0118] The acquisition module 10 is configured to acquire at least one target image from a first image set, the target image being an image of the abnormal organism segmented from a first image in the first image set, and the first image set comprising a plurality of first images.

[0119] The processing module 20 is configured to synthesize the target image with a background image to obtain the sample image, wherein the background image is obtained by photographing the forbidden area.

[0120] The functions of each module are described above in the image set generation method and will not be described here.

[0121] Figure 12 A structure diagram of an image set generation device provided by an embodiment of the present disclosure is shown in the figure. As shown in the figure, the image set generation device comprises an acquisition module 10 and a processing module 20. Figure 12 Figure 1 The computer program is executed by the processor 102 to implement the image set generation method described above, for example, to implement steps S1 to S2 in the figure.

[0122] The electronic device 100 can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing devices. The electronic device 100 can include, but is not limited to, a processor 102 and a memory 101. Those skilled in the art can understand that the electronic device 100 can further include other components, and the components of the electronic device 100 are not limited to the processor 102 and the memory 101.​​​Figure 12 The electronic device 100 is merely an example and does not limit the electronic device 100, and can include more or less components than illustrated, or combine certain components, or different components, for example, the electronic device 100 can also include an input / output device, a network access device, a bus, etc.

[0123] The processor 102 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor 102 can be a microprocessor or the processor can also be any conventional processor.

[0124] The memory 101 can be an internal storage unit of the electronic device 100, for example, a hard disk or a memory of the electronic device 100. The memory 101 can also be an external storage device of the electronic device 100, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 101 can include both the internal storage unit and the external storage device of the electronic device 100. The memory 101 is used to store the computer program and other programs and data required by the terminal device. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0126] Figure 13 A structural schematic diagram of a computer readable storage medium provided by an embodiment of the present disclosure is shown in FIG. 2. As shown in FIG. 2, the computer readable storage medium 200 stores a computer program 201, wherein the computer program 201 is executed by a processor to implement the image set generation method described above, for example, to implement steps S1 to S2 in the method. Figure 13 Figure 1 The computer readable storage medium 200 includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that a communication medium generally includes computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0127] It can be understood that the above embodiments are only exemplary embodiments adopted for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and essence of the present disclosure, and these modifications and improvements are also considered as the protection scope of the present disclosure.​

Claims

1. A method of generating a set of images, characterized by, The image set is used for training a detection model for an abnormal organism in a forbidden area, and the image set includes a plurality of sample images; the method includes generating each sample image according to the following steps: According to a first image set obtained in advance, at least one target image is obtained, the target image being an image of the abnormal organism segmented from a first image in the first image set; The target image is synthesized with a background image to obtain the sample image; The first image set includes a plurality of first images, and the background image is obtained by photographing the forbidden area; The background image is obtained by photographing the forbidden area using a target device, and the target image is synthesized with the background image to obtain the sample image, including: According to the corresponding coordinates of the to-be-pasted position, the device parameters of the target device, the image size of the background image, and the preset size of the abnormal organism, a target size parameter for synthesizing the target image into the background image is determined; the to-be-pasted position is the position of the target image in the background image when the target image is synthesized with the background image; According to the target size parameter, the target image is size-adjusted to obtain an adjusted target image; The adjusted target image is synthesized with the background image to obtain the sample image; The device parameters of the target device at least include: the installation height of the target device, the focal length of the target device, and the included angle between the optical axis of the target device and the vertical direction, According to the corresponding coordinates of the to-be-pasted position, the device parameters of the target device, the image size of the background image, and the preset size of the abnormal organism, a target size parameter for synthesizing the target image into the background image is determined, including: According to the corresponding coordinates of the to-be-pasted position and the focal length of the target device, a first angle is determined; the first angle is the included angle between the line connecting the position of the target device and the bottom position of the abnormal organism and the optical axis of the target device; According to the first angle, the installation height of the target device, the included angle between the optical axis of the target device and the vertical direction, and the preset size of the abnormal organism, a second angle is determined; wherein the preset size of the abnormal organism is determined according to the type of the abnormal organism; the second angle is the included angle between the first line and the second line, the first line being the line connecting the position of the target device and the bottom position of the abnormal organism, and the second line being the line connecting the position of the target device and the top position of the abnormal organism; According to the first angle, the second angle, and the image size, the target size parameter is determined.

2. The method of claim 1, wherein, According to a first image set obtained in advance, at least one target image is obtained, including: Each first image in the first image set is subjected to instance segmentation processing to obtain a target image set corresponding to each first image; the target image set includes at least one target image.

3. The method of claim 2, wherein, The instance segmentation processing is performed on each first image in the first image set to obtain a target image set corresponding to each first image. The first image is input into a Mask R-CNN instance segmentation network for processing to obtain the target image set.

4. The method of claim 1, wherein, Before the target size parameter is determined, the method further comprises: performing semantic segmentation processing on the background image to determine a plurality of image regions in the background image; and taking one of the plurality of image regions as a target region; determining any position in the target region as the to-be-pasted position.

5. The method of claim 4, wherein, The semantic segmentation processing on the background image to determine a plurality of image regions in the background image comprises: The background image is input into a U-Net semantic segmentation network for processing to obtain a plurality of image regions.

6. The method of claim 1, wherein, The size adjustment of the target image according to the target size parameter to obtain an adjusted target image comprises: determining an adjustment ratio of the target image according to the image size and the target size parameter; performing width adjustment and height adjustment on the target image respectively according to the adjustment ratio to obtain the adjusted target image.

7. The method of claim 1, wherein, The image synthesis of the adjusted target image and the background image to obtain the sample image comprises: determining a calibration position in the adjusted target image according to the type of the abnormal organism; pasting the adjusted target image into the background image to obtain a first image, and the calibration position in the first image is aligned with a to-be-pasted position of the background image; performing color adjustment on the first image to obtain the sample image, and the color adjustment comprises brightness adjustment and / or chroma adjustment.

8. The method of claim 7, wherein, The color adjustment on the first image to obtain the sample image comprises: setting a region in the first image outside a region where the target image is located to a first preset color to obtain a second image; inputting the first image and the second image into a color neural network to perform color adjustment on the first image to obtain the sample image.

9. An apparatus for generating a set of images, characterized by The image set is used for training a detection model for forbidden zone abnormal organisms, and the image set comprises a plurality of sample images; and the device comprises: an acquisition module configured to acquire at least one target image from a first image set according to the first image set acquired in advance, the target image being an image of the abnormal organism segmented from the first image in the first image set, and the first image set comprising a plurality of first images; a processing module configured to synthesize the target image and a background image to obtain the sample image, wherein the background image is obtained by photographing the forbidden zone; wherein the background image is obtained by photographing the forbidden zone using a target device, and the synthesis of the target image and the background image to obtain the sample image comprises: determine a target size parameter of synthesizing the target image into the background image according to the corresponding coordinate of the position to be pasted, the device parameter of the target device, the image size of the background image, and the preset size of the abnormal organism; the position to be pasted is a position of the target image in the background image when the target image is synthesized into the background image; adjust the size of the target image according to the target size parameter to obtain an adjusted target image; perform image synthesis on the adjusted target image and the background image to obtain the sample image; wherein the device parameter of the target device at least includes: an installation height of the target device, a focal length of the target device, an included angle between an optical axis of the target device and a vertical direction, determining the target size parameter of synthesizing the target image into the background image according to the corresponding coordinate of the position to be pasted, the device parameter of the target device, the image size of the background image, and the preset size of the abnormal organism, includes: determining a first angle according to the corresponding coordinate of the position to be pasted and the focal length of the target device; the first angle is an included angle between a line connecting a position of the target device and a bottom position of the abnormal organism and an optical axis of the target device; determining a second angle according to the first angle, the installation height of the target device, the included angle between the optical axis of the target device and the vertical direction, and the preset size of the abnormal organism; wherein the preset size of the abnormal organism is determined according to a type of the abnormal organism; the second angle is an included angle between a first line and a second line, the first line is a line connecting the position of the target device and the bottom position of the abnormal organism, and the second line is a line connecting the position of the target device and a top position of the abnormal organism; determining the target size parameter according to the first angle, the second angle, and the image size.

10. An image set generating apparatus comprising a memory and a processor, the memory having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

Images