Image data processing method, device and computer readable storage medium
By determining the area to be selected in the original image and using the deep residual network model to identify the appropriate location, the problem of inaccurate insertion of advertising data is solved, the display effect is improved and human resources is saved, and efficient advertising data generation is achieved.
Patent Information
- Application Number
- CN201910498450.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-06-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-07-10
AI Technical Summary
In the prior art, when generating advertising data, the inaccurate position of the billboard leads to poor display effect, and manual insertion of the billboard consumes a lot of human resources and is inefficient.
By obtaining the area to be selected in the original image, extracting the surrounding image features, determining the area type according to the service applicable evaluation parameters, and performing convolution operations in the depth residual network model to determine the target area, and then accurately inserting the service image.
It improves the accuracy of the insertion position of advertising data in the original image, improves the display effect, and saves human resources and improves the efficiency of advertising data generation.
Smart Images

Figure CN110197522B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to an image data processing method, device and computer-readable storage medium. Background Art
[0002] With the continuous development of computer networks, the network provides users with a lot of visual information, including a lot of advertising information. Due to the variety of advertising products, how to improve the display effect of advertising data generated for advertising products has put forward higher requirements.
[0003] In the prior art, a background image is selected and a billboard corresponding to the advertisement product is randomly inserted into the background image to generate advertisement data of the advertisement product. Since the billboard is randomly inserted into the background image, it is inevitable that the position of the billboard inserted into the background image is not suitable. If the position of the billboard inserted into the background image is inaccurate, the display effect of the generated advertisement data will be poor.
[0004] In addition, in the prior art, a billboard can be inserted into the background image manually, that is, a billboard is inserted into the background image by manually dragging the billboard to generate advertising data for the advertising product. However, manually generating advertising data consumes a lot of human resources, and the efficiency of generating advertising data is also low. Summary of the invention
[0005] The embodiments of the present invention provide an image data processing method, device and computer-readable storage medium, which can improve the display effect of generated advertising data, save human resources and improve the efficiency of generating advertising data.
[0006] An embodiment of the present invention provides an image data processing method, including:
[0007] Acquire an original image, and determine a plurality of areas to be selected in the original image;
[0008] Acquire a position region to which each of the multiple regions to be selected belongs in the original image; the position region includes the region to be selected, and the size of the position region is larger than the size of the region to be selected;
[0009] Determine the images in the position area except the area to be selected as surrounding images corresponding to the area to be selected;
[0010] According to the surrounding images corresponding to each of the to-be-selected areas, a target area is determined among the multiple to-be-selected areas; the target area is an area in the original image used for inserting a business image.
[0011] Wherein, determining the target area among the multiple areas to be selected according to the surrounding images corresponding to each area to be selected includes:
[0012] Extracting surrounding image features from the surrounding image, and determining a service applicability evaluation parameter corresponding to image content contained in the surrounding image according to the surrounding image features;
[0013] Determine the area type of each area to be selected according to the business applicability evaluation parameter; the area type includes a business area type and a non-business area type;
[0014] The to-be-selected area whose area type is the service area type is determined as the target area.
[0015] The service applicability evaluation parameter is a service applicability probability; and determining the area type of each area to be selected according to the service applicability evaluation parameter includes:
[0016] Determine the type of the area to be selected corresponding to the image content where the service applicability probability is greater than or equal to the probability threshold as the service area type;
[0017] The type of the area to be selected corresponding to the image content whose service applicability probability is less than the probability threshold is determined as the non-service area type.
[0018] The step of extracting surrounding image features from the surrounding image and determining the service applicability evaluation parameters corresponding to the image content contained in the surrounding image according to the surrounding image features includes:
[0019] Inputting the surrounding images corresponding to each area to be selected into the input layer of the deep residual network model;
[0020] In the deep residual network model, extracting surrounding image features in the surrounding image;
[0021] A convolution operation is performed on the surrounding image features based on the convolution block in the deep residual network model, and a business applicability evaluation parameter corresponding to the image content in the surrounding image is determined based on the convolution operation result.
[0022] The step of determining the to-be-selected area whose area type is the business area type as the target area includes:
[0023] In the to-be-selected area whose area type is the business area type, determining the to-be-selected area that meets the area aspect ratio rule as the key selection area;
[0024] The target area is determined from the key selection areas according to a first quantity threshold of the business images and a business applicability evaluation parameter corresponding to each key selection area.
[0025] Wherein, determining a plurality of areas to be selected in the original image includes:
[0026] determining a second quantity threshold for selecting a region and a region size;
[0027] The to-be-selected regions are randomly determined in the original image according to the second quantity threshold and the region size; the number of the to-be-selected regions is the second quantity threshold.
[0028] Among them, it also includes:
[0029] Acquire a sample fusion image; the sample fusion image includes a sample original image and a sample business image; the sample original image includes position mark information corresponding to the sample business image;
[0030] Sampling is performed in the sample original image according to the position mark information to obtain a positive sample sampling image and a negative sample sampling image associated with the sample business image, and an initial neural network model is trained based on the positive sample sampling image and the negative sample sampling image;
[0031] The initial neural network model trained to a converged state is determined as the deep residual network model.
[0032] The step of sampling the sample original image according to the position mark information to obtain a positive sample sample image and a negative sample sample image associated with the sample business image, and training an initial neural network model based on the positive sample sample image and the negative sample sample image includes:
[0033] Determine, according to the position mark information, a sample position area to which the sample business image belongs in the sample original image; the sample position area includes the sample business image, and a size of the sample position area is larger than a size of the sample business image;
[0034] Sampling sample surrounding images in images other than the sample business image in the sample position area, and sampling sample far-away images in images other than the sample area position in the original image;
[0035] Setting a positive sample label for the sample surrounding image, and determining the sample surrounding image with the positive sample label as the positive sample sampling image;
[0036] Setting a negative sample label for the sample far-away image, and determining the sample far-away image with the negative sample label as the negative sample sampling image;
[0037] The initial neural network model is trained based on the positive sample sampling images and the negative sample sampling images.
[0038] The step of sampling the sample original image according to the position mark information to obtain a positive sample sample image and a negative sample sample image associated with the sample business image, and training an initial neural network model based on the positive sample sample image and the negative sample sample image includes:
[0039] Obtaining the number of iterative trainings of the initial neural network model, and determining the training stages of the initial neural network model for the positive sample image and the negative sample image according to the number of iterative trainings; the training stages include a first training stage and a second training stage;
[0040] A first training phase corresponding to the initial neural network model is performed based on a first learning rate, and a second training phase corresponding to the initial neural network model is performed based on a second learning rate.
[0041] An embodiment of the present invention provides an image data processing device, including:
[0042] A first determination module is used to obtain an original image and determine a plurality of areas to be selected in the original image;
[0043] A first acquisition module is used to acquire a position area to which each of the multiple to-be-selected areas belongs in the original image; the position area includes the to-be-selected area, and the size of the position area is larger than the size of the to-be-selected area;
[0044] A second determining module, used to determine the images in the position area except the area to be selected as surrounding images corresponding to the area to be selected;
[0045] The third determination module is used to determine a target area among the multiple areas to be selected according to the surrounding images corresponding to each area to be selected; the target area is the area in the original image used for inserting the business image.
[0046] Wherein, the third determination module includes:
[0047] an extraction unit, configured to extract surrounding image features from the surrounding image, and determine a service applicability evaluation parameter corresponding to image content contained in the surrounding image according to the surrounding image features;
[0048] A first determining unit is used to determine the area type of each area to be selected according to the business applicability evaluation parameter; the area type includes a business area type and a non-business area type;
[0049] The second determining unit is configured to determine the to-be-selected area whose area type is the service area type as the target area.
[0050] Wherein, the service applicability evaluation parameter is a service applicability probability; and the image data processing device further includes:
[0051] A first type determination module, configured to determine the type of the to-be-selected area corresponding to the image content whose service applicability probability is greater than or equal to a probability threshold as the service area type;
[0052] The second type determination module is used to determine the type of the to-be-selected area corresponding to the image content whose service applicability probability is less than the probability threshold as the non-service area type.
[0053] Wherein, the extraction unit comprises:
[0054] An input subunit, used to input the surrounding image corresponding to each to-be-selected area into the input layer of the deep residual network model;
[0055] An extraction subunit, configured to extract surrounding image features from the surrounding image in the deep residual network model;
[0056] A determination subunit is used to perform a convolution operation on the surrounding image information based on the convolution block in the deep residual network model, and determine the business applicability evaluation parameters corresponding to the image content in the surrounding image based on the convolution operation result.
[0057] Wherein, the second determining unit includes:
[0058] A rule determination subunit, configured to determine, among the to-be-selected areas whose area type is the business area type, the to-be-selected areas that meet the area aspect ratio rule as key selection areas;
[0059] The applicability determination subunit is used to determine the target area from the key selection areas according to the first quantity threshold of the business images and the business applicability evaluation parameter corresponding to each key selection area.
[0060] Wherein, the first determining module includes:
[0061] a third determining unit, configured to determine a second quantity threshold for selecting an area and an area size;
[0062] A random determination unit is used to randomly determine the area to be selected in the original image according to the second quantity threshold and the area size; the number of the area to be selected is the second quantity threshold.
[0063] Wherein, the image data processing device further includes:
[0064] The second acquisition module is used to acquire a sample fusion image; the sample fusion image includes a sample original image and a sample business image; the sample original image includes position mark information corresponding to the sample business image;
[0065] A sampling module, used to sample the sample original image according to the position mark information, obtain a positive sample sample image and a negative sample sample image associated with the sample business image, and train an initial neural network model based on the positive sample sample image and the negative sample sample image;
[0066] A training module is used to determine the initial neural network model trained to a converged state as the deep residual network model.
[0067] Wherein, the sampling module comprises:
[0068] an area determination unit, configured to determine, according to the position mark information, a sample position area to which the sample business image in the sample original image belongs; the sample position area includes the sample business image, and a size of the sample position area is larger than a size of the sample business image;
[0069] A sampling unit, configured to sample a sample surrounding image in the image other than the sample business image in the sample position area, and sample a sample far-away image in the image other than the sample area position in the original image;
[0070] A first label setting unit, configured to set a positive sample label for the sample surrounding image, and determine the sample surrounding image with the positive sample label as the positive sample sampling image;
[0071] A second label setting unit is used to set a negative sample label for the sample far-away image, and determine the sample far-away image with the negative sample label as the negative sample sampling image;
[0072] The first training unit is used to train the initial neural network model based on the positive sample sampling image and the negative sample sampling image.
[0073] Wherein, the sampling module comprises:
[0074] An iterative training unit, used to obtain the number of iterative training of the initial neural network model, and determine the training stage of the initial neural network model for the positive sample sampling image and the negative sample sampling image according to the number of iterative training; the training stage includes a first training stage and a second training stage;
[0075] The second training unit is used to perform a first training phase corresponding to the initial neural network model based on a first learning rate, and to perform a second training phase corresponding to the initial neural network model based on a second learning rate.
[0076] In one aspect, the present invention provides an image data processing device, comprising: a processor and a memory;
[0077] The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program to execute a method as in one aspect of an embodiment of the present invention.
[0078] In one aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method in the above aspect.
[0079] The embodiment of the present invention first obtains the original image, determines multiple areas to be selected in the original image; obtains the position area to which each of the multiple areas to be selected belongs in the original image; the position area contains the area to be selected, and the size of the position area is larger than the size of the area to be selected; the image in the position area other than the area to be selected is determined as the surrounding image corresponding to the area to be selected; according to the surrounding images corresponding to each area to be selected, a target area is determined in the multiple areas to be selected; the target area is the area in the original image for inserting the business image. It can be seen that the method proposed in the embodiment of the present invention can first generate several areas to be selected in the original image, and then determine whether the area to be selected is suitable for inserting the business image according to the image information pointed to by the surrounding images of the area to be selected in the original image, so as to improve the accuracy of the position of the business image inserted in the original image, and then improve the display effect after the business image is inserted into the original image. At the same time, human resources are saved and the efficiency of inserting business images is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0081] Figure 1a is a schematic diagram of the architecture of a system provided by an embodiment of the present invention;
[0082] Figure 1bis a schematic diagram of a data synthesis scenario provided by an embodiment of the present invention;
[0083] Figure 2 is a flowchart of an image data processing method provided by an embodiment of the present invention;
[0084] Figure 3 is a flowchart of another image data processing method provided by an embodiment of the present invention;
[0085] Figure 4 is a schematic diagram of a convolution operation scenario provided by an embodiment of the present invention;
[0086] Figure 5 is a schematic diagram of a scene for acquiring surrounding images provided by an embodiment of the present invention;
[0087] Figure 6 is a schematic diagram of a scenario for determining a target area provided by an embodiment of the present invention;
[0088] Figure 7 It is a schematic diagram of a scenario for generating a service fusion image provided by an embodiment of the present invention;
[0089] Figure 8 It is a flowchart of a model training method provided by an embodiment of the present invention;
[0090] Figure 9a is a schematic diagram of the structure of a residual block provided by an embodiment of the present invention;
[0091] Figure 9b is a schematic diagram of the structure of another residual block provided by an embodiment of the present invention;
[0092] Fig.9c is a structural schematic diagram of a model provided by an embodiment of the present invention;
[0093] Figure 9d is a schematic diagram of a coordinate structure provided by an embodiment of the present invention;
[0094] Fig.10 is a structural schematic diagram of an image data processing device provided by an embodiment of the present invention;
[0095] Fig.11 It is a structural schematic diagram of another image data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0096] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0097] See also Figure 1a , is a schematic diagram of the architecture of a system provided by an embodiment of the present invention. Figure 1a As shown, the architecture includes a server 100 and multiple terminals. Terminal 200a, terminal 200b and terminal 200c are used as examples for explanation. The specific number and type of terminals are determined according to the actual application scenario and are not limited here. The server 100 can communicate with terminals 200a, 200b and 200c. Here, the communication between terminal 200a and server 100 is used as an example to describe the embodiment of the present invention. The server 100 can generate a business fusion image based on the acquisition request for the business fusion image sent by terminal 200a, and send the generated business fusion image to terminal 200a, or the server can spontaneously send the generated business fusion image to terminal 200a. Terminal 200a can display the business fusion image sent by server 100 in the terminal interface. For example, when the above-mentioned business fusion image is an advertising data image corresponding to an advertising product, terminal 200a can display the advertising data image in a web page interface or an application interface. Among them, the specific process of the server 100 generating the business fusion image can be referred to as follows Figure 1b The process described.
[0098] See also Figure 1b , is a schematic diagram of a data synthesis scenario provided by an embodiment of the present invention. Figure 1b As shown, first, the server 100 can generate multiple areas to be selected in the original image a10. Here, the generation of three areas to be selected is taken as an example for explanation. The three areas to be selected are area a1, area a2 and area a3. The specific number of areas to be selected is determined according to the actual application scenario and is not limited here. Next, the server 100 obtains the surrounding images corresponding to the above-mentioned areas a1, a2 and a3 respectively. The specific process is to obtain the position area corresponding to each area to be selected in the original image, the position area contains the area to be selected, and the size of the position area is larger than the area to be selected, and the image in the position area except the area to be selected is used as the surrounding image corresponding to the area to be selected. Figure 1bAs shown, the server 100 obtains the surrounding image of the above-mentioned area a1 as image a4, obtains the surrounding image of the above-mentioned area a2 as image a5, and obtains the surrounding image of the above-mentioned area a3 as image a6. The server 100 inputs the above-mentioned images a4, a5 and a6 into the context model a7, and the image content contained in the images a4, a5 and a6 can be identified through the context model a7. The business applicability evaluation parameters corresponding to the images a4, a5 and a6 are obtained through the identification results. The business applicability evaluation parameters can be business applicability probabilities. If the value of the business applicability probability corresponding to a certain surrounding image is larger, it indicates that the type of the area to be selected corresponding to the surrounding image is more inclined to the business area type, that is, it is more suitable for inserting business images. On the contrary, if the value of the business applicability probability corresponding to a certain surrounding image is smaller, it indicates that the type of the area to be selected corresponding to the surrounding image is more inclined to the non-business area type, that is, it is less suitable for inserting business images. The target area can be selected from the above-mentioned area a1, area a2 and area a3 according to the business applicability evaluation parameters corresponding to the images a4, a5 and a6 respectively. For example, the target area is selected according to the probability values of the business applicability probabilities corresponding to the images a4, a5 and a6 respectively. Specifically, a probability threshold can be set, and the area corresponding to the image whose business applicability probability is greater than or equal to the probability threshold in the images a4, a5 and a6 is selected as the target area. Here, the area a1 is selected as the target area as an example. The set a8 is a business image data set, and the set a8 includes several images, which may include advertising images, poster images and photo frame images. The business image can be obtained in the set a8, and the business image can be inserted into the position where the target area is located in the original image, that is, the obtained business image is inserted into the area a1 in the original image a10 to obtain the business fusion image a9. Among them, the above-mentioned original image can be a background image, the above-mentioned business image can be a billboard, and the above-mentioned business fusion image can be an advertising data image synthesized by inserting a billboard into the background image. The above-mentioned context model a7 can be a deep residual network model.
[0099] The method provided by the embodiment of the present invention is adopted to obtain multiple areas to be selected in the original image in advance (the multiple areas to be selected can be multiple frames), identify the image content contained in the surrounding image of each selected area to be selected through the context model, and obtain the business applicability evaluation parameters corresponding to the identified image content. Then, a target area is selected from the above multiple areas to be selected according to the business applicability evaluation parameters corresponding to each area to be selected, and a business image is inserted into the target area to synthesize a business fusion image. The method provided by the embodiment of the present invention can effectively improve the accuracy of the position where the business image is inserted in the original image, thereby effectively improving the display effect of the synthesized business fusion image.
[0100] See also Figure 2 , is a flow chart of an image data processing method provided by an embodiment of the present invention, such as Figure 2 As shown, the method may include:
[0101] Step S101, obtaining an original image, and determining a plurality of areas to be selected in the original image;
[0102] Specifically, the original image is obtained, and the original image can be a photo taken in real time, or a background image obtained from a database, or an image crawled on the Internet. Several areas to be selected are randomly generated in the original image, and the size and number of the areas to be selected can be set. For example, the size range of the randomly generated areas to be selected can be set to 30*30 to 50*50, and the size of the areas to be selected can be 30*30, 30*40, 40*50, etc. Or the length range of the areas to be selected is set to [30,60], and the width range is set to [30,50], and the areas to be selected can be 40*60, 30*40, 40*50, etc. The number of the above-mentioned areas to be selected can also be set, for example, 200 areas to be selected are randomly generated in the original image, and the generated areas to be selected contain part of the image content of the original image. The above-mentioned areas to be selected can be in the shape of a rectangular frame, a circular frame or a trapezoidal frame. The specific shape of the frame corresponding to the area to be selected is determined according to the actual application scenario, and is not limited here. The shape and size of each of the multiple areas to be selected may be the same or different.
[0103] Step S102, obtaining a position region to which each of the plurality of regions to be selected belongs in the original image; the position region includes the region to be selected, and the size of the position region is larger than the size of the region to be selected;
[0104] Specifically, the position area of each of the above-mentioned areas to be selected in the original image is obtained, and the position area is generated with the above-mentioned area to be selected as the center. The above-mentioned area to be selected is contained in the position area, and the size of the position area is larger than the size of the area to be selected. The position area can be a frame generated with the above-mentioned area to be selected as the center and on the periphery of the area to be selected. For example, the size of the area to be selected is 30*30, and the area to be selected can be taken as the center and the areas expanded outward by 10 on all sides are used as the position area of the area to be selected. The specific shape of the frame corresponding to the position area can be determined according to the shape of the area to be selected, and there is no limitation here. For example, the shape of the frame corresponding to the position area can be the same as the shape of the frame corresponding to the area to be selected. Among them, the processing method of each area to be selected is parallel and the same. For the convenience of description, the description of the embodiment of the present invention is described by taking the processing process corresponding to an area to be selected as an example.
[0105] Step S103, determining the images in the position area except the area to be selected as the surrounding images corresponding to the area to be selected;
[0106] Specifically, the image in the position area except the area to be selected is used as the surrounding image corresponding to the area to be selected. For example, the image in the area to be selected is dug out in the position area, and the remaining image in the position area after the image in the area to be selected is used as the surrounding image corresponding to the area to be selected. The environment of the area to be selected can be judged by the surrounding image, for example, the area to be selected is on a wall, in a river, or on a bench, etc.
[0107] Step S104, determining a target area among the multiple areas to be selected according to the surrounding images corresponding to each area to be selected; the target area is an area in the original image for inserting a business image;
[0108] Specifically, according to the surrounding images corresponding to the above-mentioned area to be selected, the target area is determined in the above-mentioned multiple areas to be selected. The surrounding images corresponding to the obtained area to be selected are input into the context model, and the specific image content contained in the surrounding images corresponding to the area to be selected is identified by the context model. For example, if it is identified that the above-mentioned surrounding images contain rivers, walls, trees and windows, etc., then it can be judged that walls and windows are more suitable for inserting business images, and rivers and trees are not suitable for inserting business images. That is, the area to be selected corresponding to the above-mentioned walls and windows can be selected as the target area. The business image is inserted into the position selected as the target area in the multiple areas to be selected generated in the above-mentioned original image. The business image can be an image such as a billboard, a photo frame or a poster. By inserting the business image in the target area in the original image, a business fusion image can be obtained, that is, an image obtained by fusing the business image and the original image. The above-mentioned generated business fusion image can be an advertising image, and the above-mentioned generated advertising image can be displayed in a web page or application, for example, in the form of a pop-up window.
[0109] Furthermore, when inserting a business image in the target area, the specific image content contained in the surrounding image of the identified target area can be matched with the labels of all business images contained in the business image data set. That is, according to the label of each business image, a business image matching the image feature of the image content contained in the surrounding image of the identified target area is selected in the business image data set. The selected business image matching the surrounding image is inserted into the target area corresponding to the surrounding image. For example, when the surrounding image of the matched target area contains a river or a tree, a business image with a label of "environmental protection" can be matched in the business image data set and inserted into the target area. For another example, when the surrounding image of the matched target area contains furniture, a business image with a label of "photo frame", "poster", "oil painting" or "decorative painting" can be matched in the business image data set and inserted into the target area. There can also be a matching degree between the image content in a surrounding image and the label of the business image in the business image data set. When the labels of multiple business images are matched to be consistent with the image content of the surrounding image of the target area, the business image to be inserted into the target area can be selected from the multiple matched business images according to the matching degree corresponding to each matched business image. For example, a business image corresponding to the highest matching degree value is selected and inserted into the target area.
[0110] The embodiment of the present invention first obtains the original image, determines multiple areas to be selected in the original image; obtains the position area to which each of the multiple areas to be selected belongs in the original image; the position area contains the area to be selected, and the size of the position area is larger than the size of the area to be selected; the image in the position area other than the area to be selected is determined as the surrounding image corresponding to the area to be selected; according to the surrounding images corresponding to each area to be selected, a target area is determined in the multiple areas to be selected; the target area is the area in the original image for inserting the business image. It can be seen that the method proposed in the embodiment of the present invention can first generate several areas to be selected in the original image, and then determine whether the area to be selected is suitable for inserting the business image according to the image information pointed to by the surrounding images of the area to be selected in the original image, so as to improve the accuracy of the position of the business image inserted in the original image, and then improve the display effect after the business image is inserted into the original image. At the same time, human resources are saved and the efficiency of inserting business images is improved.
[0111] See also Figure 3 , is a flow chart of another image data processing method provided by an embodiment of the present invention, such as Figure 3 As shown, the method may include:
[0112] Step S201, determining a second quantity threshold and a region size for selecting a region;
[0113] Specifically, determine the second quantity threshold and region size for selecting the region. For example, the region size of the region to be selected that needs to be generated can be set to 30*30 to 50*50, and the size of the region to be selected can be 30*30, 30*40, 40*50, etc. Or the length range of the region to be selected can be set to [30,60], and the width range can be set to [30,50], and the region to be selected can be 40*60, 30*40, 40*50, etc. The number of regions to be selected that need to be generated (i.e., the second quantity threshold) can also be set. For example, if the number of regions to be selected is set to 200, it means that 200 regions to be selected need to be generated in the original image.
[0114] Step S202, randomly determining the to-be-selected regions in the original image according to the second quantity threshold and the region size; the number of the to-be-selected regions is the second quantity threshold;
[0115] Specifically, the to-be-selected regions are randomly determined in the original image according to the second quantity threshold and the region size. For example, according to the region size of the to-be-selected regions set as above, to-be-selected regions whose number is the second quantity threshold are generated in the original image.
[0116] Step S203, obtaining a location area in the original image to which each of the multiple to-be-selected areas belongs;
[0117] Step S204, determining the images in the position area except the area to be selected as the surrounding images corresponding to the area to be selected;
[0118] The specific implementation process of the above steps S203 to S204 can be found in the above Figure 2 The description of step S102 to step S103 in the corresponding embodiment will not be repeated here.
[0119] Step S205, inputting the surrounding images corresponding to each area to be selected into the input layer of the deep residual network model;
[0120] Specifically, the surrounding images corresponding to each area to be selected obtained above are input into the input layer of the context model, wherein the context model may be ResNet50 (i.e., a deep residual network model). That is, the surrounding images corresponding to each area to be selected are input into the input layer of ResNet50. ResNet50 has a total of 50 layers, including 5 convolutional blocks, which can effectively recognize and classify the input images.
[0121] Step S206, extracting surrounding image features in the surrounding image in the deep residual network model;
[0122] Specifically, in the above-mentioned deep residual network model, the surrounding image features in the above-mentioned surrounding images are extracted, for example, a feature map corresponding to each surrounding image is obtained by extracting the image information, and the feature map is used to characterize the image feature information contained in the surrounding images, and includes the image features contained in the surrounding images. In the field of image processing, the features contained in the image are usually characterized by the feature map of the image, and the feature map can be in the form of a matrix.
[0123] Step S207, performing a convolution operation on the surrounding image features based on the convolution block in the deep residual network model, and determining a business applicability evaluation parameter corresponding to the image content in the surrounding image based on the convolution operation result;
[0124] Specifically, the convolution operation is performed on the features of the surrounding images through the convolution blocks in the deep residual network model. For example, the convolution operation is performed on the feature map corresponding to the surrounding images through the convolution blocks in the deep residual network model. The image content in the surrounding images is identified through the convolution operation results and the recognition results of each level of the model, for example, the image content is a river, a wall, a mirror or a window. And the business applicability evaluation parameters corresponding to the identified image content are obtained, that is, the business applicability evaluation parameters of the area to be selected corresponding to the image content, which can be obtained by outputting the fully connected layer in the deep residual network model. For example, the business applicability evaluation probability corresponding to each area to be selected is obtained.
[0125] See also Figure 4 , is a scene diagram of a convolution operation provided by an embodiment of the present invention. Through multiple convolution layers in the deep residual network model, convolution operations are performed on the feature maps corresponding to the above-mentioned surrounding images respectively to obtain image feature information with image content attributes. The size of the surrounding image features can be expressed as C×H×W, or the surrounding image features can be understood as a combination of C unit original image feature information with a height of H and a width of W.
[0126] Each convolution layer corresponds to one or more convolution kernels (also called filters or receptive fields). The number of channels of the convolution kernel in each convolution coding layer is determined by the number of channels of the input data. The number of channels of the output data (that is, image feature information) of each layer is determined by the number of convolution kernels in the coding convolution layer, and the image height H in the output data is out and image width W out (i.e., the second and third dimensions in the output data) are determined by the size of the input data, the size of the convolution kernel, the stride, and the padding, i.e., H out =(H in-H kernel +2*padding) / stride+1,W out =(W in -W kernel +2*padding) / stride+1. H in ,H kernel Respectively represent the height of the image in the input data and the height of the convolution kernel; W in ,W kernel They represent the width of the image in the input data and the width of the convolution kernel respectively.
[0127] If the specific size of the surrounding image features in the above is 1×3×3 (the first dimension is used to identify the number of channels, the second dimension is used to identify the image height, and the third dimension is used to identify the image width), it is a 3×3 matrix with 1 channel; the convolution kernel has a size of 1×2×2 (the first dimension is used to identify the number of channels, the second dimension is used to identify the height of the convolution kernel, and the third dimension is used to identify the width of the convolution kernel), which is a 2×2 matrix with 1 channel.
[0128] Assume stride = 1, padding = 0, the specific process of convolution operation is: first, the value in the dotted box in matrix 10a is multiplied by the convolution kernel, that is: 1*1+0*(-1)+1*1+1*(-1)=1, and the matrix 10e can be determined;
[0129] Then, the value in the dotted box in the matrix 10b is multiplied by the convolution kernel, that is, 0*1+1*(-1)+1*1+1*(-1)=-1; the matrix 10f can be determined;
[0130] Then, the value in the dotted box in matrix 10c is multiplied by the convolution kernel, that is, 1*1+1*(-1)+1*1+0*(-1)=1; matrix 10g can be determined;
[0131] Finally, the value in the dotted box in matrix 10d is multiplied by the convolution kernel, that is, 1*1+1*(-1)+0*1+1*(-1)=-1; matrix 10h can be determined. Matrix 10h is the surrounding image feature information with a size of 1×2×2.
[0132] Step S208, determining the area type of each area to be selected according to the service applicability evaluation parameter; the area type includes a service area type and a non-service area type;
[0133] Specifically, the area type of each area to be selected is determined according to the above-mentioned business applicability evaluation parameter. For example, when the above-mentioned business applicability evaluation parameter is the business applicability probability, a probability threshold can be set, and the type of the area to be selected whose business applicability probability is greater than or equal to the probability threshold is determined as a business area type, that is, an area type that is not suitable for inserting a business image. The type of the area to be selected whose business applicability probability is less than the probability threshold is determined as a non-business area type, that is, an area type that is suitable for inserting a business image.
[0134] Step S209, determining the to-be-selected area whose area type is the business area type as the target area;
[0135] Specifically, the above-mentioned area to be selected whose area type is a business area type is determined as the target area. Among them, all the areas to be selected whose area type is a business area type can be used as the target area, and the number of business images to be inserted can also be set to select the target area in the area to be selected whose area type is a business area type. For example, when the number of business images to be inserted is 3, three areas to be selected can be selected as the target area in the area to be selected whose area type is a business area type according to the business applicability evaluation parameter corresponding to each area to be selected. For example, when the business applicability evaluation parameter is the business applicability probability, the three areas to be selected with the highest business applicability probability can be selected as the target area in the area to be selected whose area type is a business area type according to the numerical value of the business applicability probability. It is also possible to select the area to be selected that meets the area aspect ratio rule in the area to be selected whose area type is the business area type as the target area, that is, select the area to be selected whose length-width ratio meets the rule in the area to be selected whose area type is the business area type as the target area. Among them, the above-mentioned area aspect ratio rule can be that the aspect ratio of the limited area is greater than 0.2 and less than 0.4. It is also possible to select a key selection area from the to-be-selected area whose area type is the business area type according to the aspect ratio rule, and then select a target area from the key selection area according to the number of business images that need to be inserted (which can be called the first quantity threshold) and the business applicability probability corresponding to each key selection area.
[0136] Furthermore, the area size of the area to be selected that needs to be generated may not be set in advance, and multiple areas to be selected may be randomly generated in the original image. The surrounding images corresponding to the generated multiple areas to be selected are input into the deep residual network model to obtain the business applicability evaluation parameters corresponding to each area to be selected. According to the business applicability evaluation parameters corresponding to each area to be selected, the area to be selected of the business type is selected, and then the area size and aspect ratio rule of the area to be selected of the business image are inserted as needed, and the target area is selected from the area to be selected of the business type.
[0137] Step S210, inserting a business image into the target area in the original image to obtain a business fusion image;
[0138] The specific implementation process of step S210 can be found in Figure 2 The description of step S104 in the corresponding embodiment will not be repeated here.
[0139] The embodiment of the present invention first obtains the original image, determines multiple areas to be selected in the original image; obtains the position area to which each of the multiple areas to be selected belongs in the original image; the position area contains the area to be selected, and the size of the position area is larger than the size of the area to be selected; the image in the position area other than the area to be selected is determined as the surrounding image corresponding to the area to be selected; according to the surrounding images corresponding to each area to be selected, a target area is determined in the multiple areas to be selected; the target area is the area in the original image for inserting the business image. It can be seen that the method proposed in the embodiment of the present invention can first generate several areas to be selected in the original image, and then determine whether the area to be selected is suitable for inserting the business image according to the image information pointed to by the surrounding images of the area to be selected in the original image, so as to improve the accuracy of the position of the business image inserted in the original image, and then improve the display effect after the business image is inserted into the original image. At the same time, human resources are saved and the efficiency of inserting business images is improved.
[0140] See also Figure 5 , is a schematic diagram of a scene for obtaining surrounding images provided by an embodiment of the present invention. Figure 5 As shown, first, in the original image S13, four areas to be selected are generated, namely area s1, area s2, area s3 and area s4. Then, in the original image s14 where the areas to be selected have been generated, the position area corresponding to each area to be selected is determined, wherein the position area corresponding to area s1 is area s5, the position area corresponding to area s2 is area s6, the position area corresponding to area s3 is area s7, and the position area corresponding to area s4 is area s8. Set s15 is a set of surrounding images of the above-mentioned areas to be selected, that is, the surrounding images of the areas to be selected are removed from the position areas. Among them, the surrounding image corresponding to area s1 is image s9, the surrounding image corresponding to area s2 is image s10, the surrounding image corresponding to area s3 is image s11, and the surrounding image corresponding to area s4 is image s12. Please refer to Figure 6 , is a schematic diagram of a scenario for determining a target area provided by an embodiment of the present invention. Figure 6 As shown, set s15 is the above Figure 4The set s15 obtained in the above example contains images s9, s10, s11 and s12. Images s9, s10, s11 and s12 are input into the deep residual network model y8 to obtain the service applicability probability P1 corresponding to image s9, the service applicability probability P2 corresponding to image s12, the service applicability probability P3 corresponding to image s10 and the service applicability probability P4 corresponding to image s11. The surrounding images (here, images s9 and s10 included in set c1 are selected) corresponding to the two service applicability probabilities with the highest probability values among the above probabilities P1, P2, P3 and P4 can be selected as the target area (here, the above regions s1 and s2 included in set c2 are selected). Please refer to Figure 7 , is a schematic diagram of a scenario for generating a business fusion image provided by an embodiment of the present invention. The set g1 includes the business image g2 and the business image g3 to be inserted, and the image g2 and the image g3 are inserted into the above-selected target area s1 and the target area s2 to obtain the business fusion image g4, that is, the business fusion image g4 is obtained through the original image s13.
[0141] See also Figure 8 , is a flow chart of a model training method provided by an embodiment of the present invention, such as Figure 8 As shown, the method may include:
[0142] Step S301, obtaining a sample fusion image; the sample fusion image includes a sample original image and a sample business image; the sample original image includes position mark information corresponding to the sample business image;
[0143] Specifically, a sample fusion image is obtained. The sample fusion image can be three types of data: billboards, photo frames, and posters in the openimage v4 (image dataset) open source dataset, or it can be an image crawled from the Internet. Among them, the sample fusion image contains a sample business image and a sample original image. The sample business image may include a billboard, a photo frame, or a poster, and the position of the sample business image in the sample original image has been marked, that is, the above-mentioned billboard, photo frame, or poster corresponds to a position mark information in the sample original image. Among them, there are multiple sample fusion images, and the processing process of each sample fusion image is the same. For the convenience of description, the processing process of a sample fusion image is described as an example.
[0144] Step S302, determining a sample position region to which the sample business image in the sample original image belongs according to the position mark information; the sample position region includes the sample business image, and the size of the sample position region is larger than the size of the sample business image;
[0145] Specifically, the sample position area of the sample business image can be obtained in the sample original image according to the position mark information, the sample position area includes the sample business image, and the size of the sample position area is larger than the size of the sample business image. Figure 8 As shown, for example, according to the position mark information, the sample business image y3 is obtained to which the sample position area y2 in the original image y1 belongs.
[0146] Step S303, sampling sample surrounding images in the images other than the sample business image in the sample position area, and sampling sample far-away images in the images other than the sample area position in the original image;
[0147] Specifically, a sample surrounding image is sampled in an image other than the sample business image in the sample position area, for example, a sample surrounding image is sampled in an image other than the sample business image y3 in the sample position area y2, to obtain a sample surrounding image y5. A sample away image is sampled in an image other than the sample area position in the original image, for example, a sample away image is sampled in an image other than the sample area position y2 in the original image y1, to obtain a sample away image y4. The above-mentioned sample fusion image may also be an image not containing the sample business image, and an image may be sampled in a sample fusion image not containing the sample business image as a negative sample sampling image. In the sampling process, the image used for model training may be sampled at a sampling rate of three times that of the positive sample sampling image. The specific sampling ratio of the positive sample sampling image to the negative sample sampling image is determined according to the actual application scenario, and is not limited here.
[0148] Step S304, setting a positive sample label for the sample surrounding image, and determining the sample surrounding image with the positive sample label as the positive sample sampling image; setting a negative sample label for the sample far away image, and determining the sample far away image with the negative sample label as the negative sample sampling image;
[0149] Specifically, a positive sample label is set for the sample surrounding image, and the sample surrounding image with the positive sample label is determined as the positive sample sampling image. The image content contained in the positive sample sampling image points to the image content pointed to by the surrounding image suitable for inserting the business image. For example, a positive sample label is set for the sample surrounding image y5 to obtain the positive sample sampling image y7. A negative sample label is set for the sample far-away image, and the sample far-away image with the negative sample label is determined as the negative sample sampling image. The image content contained in the negative sample sampling image points to the image content pointed to by the surrounding image not suitable for inserting the business image. For example, a negative sample label is set for the sample far-away image y4 to obtain the negative sample sampling image y6.
[0150] Step S305, training the initial neural network model based on the positive sample sampling image and the negative sample sampling image; determining the initial neural network model trained to a converged state as the deep residual network model;
[0151] Specifically, the positive sample sampling image and the negative sample sampling image obtained above are input into the initial neural network model to train the initial neural network model. Wherein, the above initial neural network model can be a ResNet50 model trained on the ImageNet (image data set) data set, and the ResNet50 model trained on the ImageNet data set is initialized. In the method provided in the embodiment of the present invention, the ResNet50 model trained on the ImageNet data set is trained twice. The batch-size (batch size) can be used for training with a size of 32, and the Adam optimization algorithm is used to train the above initial neural network model. The number of iterative training of the model can be set to 10,000 times, and the first 8,000 times are used as the first training stage of training, and the latter 2,000 times are used as the second training stage of training. In the first training stage, the first learning rate is 0.0001 for training, and the second learning rate is used for training in the second training stage. During the second training stage, the first learning rate can be reduced by 10 times every 500 iterations of training as the second learning rate for training. That is, after the model is iterated and trained 8,000 times, the learning rate is reduced by 10 times for every 500 iterations of training. The initial neural network model is continuously trained by continuously sampling the positive sample images and the negative sample images, and the model parameters are continuously corrected during the training process until the initial neural network model is trained to a convergence state. The initial neural network model trained to a convergence state is determined as the above-mentioned deep residual network model y8 (i.e., context model). The obtained deep residual network model y8 can be used to identify the image content contained in the input image and the business applicability evaluation parameters corresponding to the image content, such as the business applicability probability. That is, the obtained deep residual network model y8 can be used to identify the image content contained in the input surrounding image and output the business applicability evaluation parameters corresponding to the identified image content.
[0152] See also Figure 9a , is a schematic diagram of the structure of a residual block provided by an embodiment of the present invention. The deep residual network model ResNet is composed of a plurality of residual blocks. Figure 9aAs shown in the figure, a two-layer residual block m1 is provided, whose input is x. The shortcut connection spans the weight layer q1 and the weight layer q2. The input x can be used as the output of the subsequent layer through the shortcut connection. Here, the output H(x) is H(x)=F(x)+x, so that when the model is trained, it is no longer directly training the input and output, but training the residual F(x). The purpose of training is to make F(x) approach 0. During the residual training process, the input is activated by the activation function between the layers to obtain the output. Please refer to Figure 9b , is a schematic diagram of the structure of another residual block provided by an embodiment of the present invention. Figure 9b As shown in FIG. 1 , a three-layer residual block is provided, wherein the implementation principle of the three-layer residual block is consistent with that of the two-layer residual block. Figure 9b As shown in , the shortcut connection of residual block m2 is a solid line, indicating that the residual F(x) has the same channel as the input x, and the shortcut connection of residual block m3 is a dotted line, indicating that the residual F(x) has a different channel than the input x. Fig.9c , is a structural diagram of a model provided by an embodiment of the present invention, namely, a deep residual network model ResNet composed of several three-layer residual blocks. Among them, the more layers of residual blocks, the less calculation and parameters of the model can be reduced. In actual application, when the deep residual network model ResNet has fewer layers, a residual block with fewer layers, such as a two-layer residual block, can be used; when the deep residual network model ResNet has more layers, a residual block with more layers, such as a three-layer residual block, can be used. If ResNet50 is used in an embodiment of the present invention, ResNet50 can be built using three-layer residual blocks. The number of layers of residual blocks for building the ResNet model is determined according to the actual application scenario and is not limited here. Among them, by using residuals to train the model, it can be made that when the number of model layers continues to increase, the error of the model becomes smaller, thereby improving the model training effect, and is suitable for deep training of the model. Please refer to Figure 9d , is a schematic diagram of a coordinate structure provided by an embodiment of the present invention, through Figure 9d From the coordinate m4 in , we can see that when the residual is used to train the model, as the number of model layers increases, the training error of the model becomes smaller.
[0153] Among them, the specific process of residual operation can be: the number of residual layers in the model can be one or more, and the dimension of the input data of each residual layer is the same as the dimension of the output data. Therefore, after the conditional convolution tensor with a size of C×H×W is residually connected through the residual layer in the ResNet model, the size of the obtained conditional transformation tensor is still C×H×W.
[0154] Residual connection means that after two consecutive convolution operations, the scope of the activation function is the sum of the output after the convolution operation and the input before the convolution operation. The calculation formula of residual connection is as follows (1):
[0155]
[0156] Among them, R out is the output of the residual layer; x is the input of the residual layer; W 1 ,W 2 They represent the weights in the first convolution operation and the second convolution operation respectively; σ represents the activation function.
[0157] According to the mask convolution layer in the ResNet model, a convolution operation is performed on the conditional transformation tensor to obtain a mask image for identifying the transformation area where the first object attribute is located, and the size of the mask image is H×W. The specific process of the convolution operation can refer to the description in the aforementioned step S207.
[0158] See also Fig.10 , is a schematic diagram of the structure of an image data processing device provided by an embodiment of the present invention. Fig.10 As shown, the image data processing device 1 may include: a first determination module 101, a first acquisition module 102, a second determination module 103 and a third determination module 104;
[0159] A first determining module 101 is used to obtain an original image and determine a plurality of areas to be selected in the original image;
[0160] A first acquisition module 102 is used to acquire a position region to which each of the multiple to-be-selected regions belongs in the original image; the position region includes the to-be-selected region, and the size of the position region is larger than the size of the to-be-selected region;
[0161] A second determining module 103, configured to determine images in the location area except the area to be selected as surrounding images corresponding to the area to be selected;
[0162] The third determination module 104 is used to determine a target area among the multiple areas to be selected according to the surrounding images corresponding to each area to be selected; the target area is the area in the original image for inserting the business image.
[0163] The specific functional implementation of the first determination module 101, the first acquisition module 102, the second determination module 103 and the third determination module 104 can be found in Figure 2 Steps S101 to S104 in the corresponding embodiment will not be described in detail here.
[0164] The third determination module 104 includes an extraction unit 1041, a first determination unit 1042, and a second determination unit 1043:
[0165] An extraction unit 1041 is used to extract surrounding image features from the surrounding image, and determine a service applicability evaluation parameter corresponding to the image content contained in the surrounding image according to the surrounding image features;
[0166] A first determining unit 1042 is used to determine the area type of each area to be selected according to the service applicability evaluation parameter; the area type includes a service area type and a non-service area type;
[0167] The second determining unit 1043 is configured to determine the to-be-selected area whose area type is the service area type as the target area.
[0168] The specific functional implementation of the extraction unit 1041 and the first determination unit 1042 can be found in Figure 3 For the specific implementation of the steps S206 to S208 in the corresponding embodiment, please refer to Figure 3 The corresponding step S209 in the embodiment will not be described in detail here.
[0169] The service applicability evaluation parameter is a service applicability probability; the image data processing device 1 further includes a first type determination module 105 and a second type determination module 106:
[0170] A first type determination module 105, configured to determine the type of the area to be selected corresponding to the image content whose service applicability probability is greater than or equal to a probability threshold as the service area type;
[0171] The second type determination module 106 is configured to determine the type of the to-be-selected area corresponding to the image content whose service applicability probability is less than the probability threshold as the non-service area type.
[0172] The specific functional implementation of the first type determination module 105 and the second type determination module 106 can be found in Figure 3 The corresponding step S208 in the embodiment will not be described in detail here.
[0173] The extracting unit 1041 includes an input subunit 10411, an extracting subunit 10412 and a determining subunit 10413:
[0174] An input subunit 10411 is used to input the surrounding image corresponding to each area to be selected into the input layer of the deep residual network model;
[0175] An extraction subunit 10412, configured to extract surrounding image features from the surrounding image in the deep residual network model;
[0176] The determination subunit 10413 is used to perform a convolution operation on the surrounding image features based on the convolution block in the deep residual network model, and determine the business applicability evaluation parameters corresponding to the image content in the surrounding image based on the convolution operation result.
[0177] The specific functional implementation of the input subunit 10411, the extraction subunit 10412 and the determination subunit 10413 can be found in Figure 3 Steps S205 to S207 in the corresponding embodiment will not be described in detail here.
[0178] The second determination unit 1043 includes a rule determination subunit 10431 and an applicability determination subunit 10432:
[0179] A rule determination subunit 10431 is used to determine, among the to-be-selected areas whose area type is the business area type, the to-be-selected areas that meet the area aspect ratio rule as key selection areas;
[0180] The applicability determination subunit 10432 is used to determine the target area from the key selection areas according to the first quantity threshold of the business images and the business applicability evaluation parameter corresponding to each key selection area.
[0181] The specific functional implementation of the rule determination subunit 10431 and the applicable determination subunit 10432 can be found in Figure 3 The corresponding step S209 in the embodiment will not be described in detail here.
[0182] The first determination module 101 includes a third determination unit 1011 and a random determination unit 1012:
[0183] The third determining unit 1011 is used to determine a second quantity threshold and a region size for selecting a region;
[0184] The random determination unit 1012 is used to randomly determine the to-be-selected area in the original image according to the second quantity threshold and the area size; the number of the to-be-selected areas is the second quantity threshold.
[0185] The specific functional implementation of the third determination unit 1011 and the random determination unit 1012 can be found in Figure 3 Steps S201 and S202 in the corresponding embodiment will not be described in detail here.
[0186] The image data processing device 1 further includes a second acquisition module 107, a sampling module 108 and a training module 109:
[0187] The second acquisition module 107 is used to acquire a sample fusion image; the sample fusion image includes a sample original image and a sample business image; the sample original image includes position mark information corresponding to the sample business image;
[0188] A sampling module 108 is used to sample the sample original image according to the position mark information to obtain a positive sample sample image and a negative sample sample image associated with the sample business image, and train an initial neural network model based on the positive sample sample image and the negative sample sample image;
[0189] The training module 109 is used to determine the initial neural network model trained to a converged state as the deep residual network model.
[0190] The specific functional implementation of the second acquisition module 107 can be found in Figure 8 For the specific implementation of the sampling module 108 in step S301 in the corresponding embodiment, please refer to Figure 8 For the specific implementation of the functions of the training module 109 in the corresponding embodiment of step S302-step S304, please refer to Figure 8 The corresponding step S305 in the embodiment will not be described in detail here.
[0191] The sampling module 108 includes a region determination unit 1081, a sampling unit 1082, a first label setting unit 1083, a second label setting unit 1084 and a first training unit 1085:
[0192] The area determination unit 1081 is used to determine the sample position area to which the sample business image in the sample original image belongs according to the position mark information; the sample position area includes the sample business image, and the size of the sample position area is larger than the size of the sample business image;
[0193] A sampling unit 1082 is used to sample sample surrounding images in the image other than the sample business image in the sample position area, and sample sample far-away images in the image other than the sample area position in the original image;
[0194] A first label setting unit 1083 is used to set a positive sample label for the sample surrounding image, and determine the sample surrounding image with the positive sample label as the positive sample sampling image;
[0195] A second label setting unit 1084 is used to set a negative sample label for the sample far-away image, and determine the sample far-away image with the negative sample label as the negative sample sampling image;
[0196] The first training unit 1085 is used to train the initial neural network model based on the positive sample sampling image and the negative sample sampling image.
[0197] The specific functional implementation of the region determination unit 1081, the sampling unit 1082, the first label setting unit 1083, the second label setting unit 1084 and the first training unit 1085 can be found in Figure 8 Steps S302 to S304 in the corresponding embodiment will not be described in detail here.
[0198] The sampling module 108 includes an iterative training unit 1086 and a second training unit 1087:
[0199] An iterative training unit 1086 is used to obtain the number of iterative training of the initial neural network model, and determine the training stage of the initial neural network model for the positive sample image and the negative sample image according to the number of iterative training; the training stage includes a first training stage and a second training stage;
[0200] The second training unit 1087 is used to perform a first training phase corresponding to the initial neural network model based on a first learning rate, and to perform a second training phase corresponding to the initial neural network model based on a second learning rate.
[0201] The specific functional implementation of the iterative training unit 1086 and the second training unit 1087 can be found in Figure 8 The corresponding step S305 in the embodiment will not be described in detail here.
[0202] The embodiment of the present invention first obtains the original image, determines multiple areas to be selected in the original image; obtains the position area to which each of the multiple areas to be selected belongs in the original image; the position area contains the area to be selected, and the size of the position area is larger than the size of the area to be selected; the image in the position area other than the area to be selected is determined as the surrounding image corresponding to the area to be selected; according to the surrounding images corresponding to each area to be selected, a target area is determined in the multiple areas to be selected; the target area is the area in the original image for inserting the business image. It can be seen that the method proposed in the embodiment of the present invention can first generate several areas to be selected in the original image, and then determine whether the area to be selected is suitable for inserting the business image according to the image information pointed to by the surrounding images of the area to be selected in the original image, so as to improve the accuracy of the position of the business image inserted in the original image, and then improve the display effect after the business image is inserted into the original image. At the same time, human resources are saved and the efficiency of inserting business images is improved.
[0203] See also Fig.11 , is a schematic diagram of the structure of another image data processing device provided by an embodiment of the present invention. Fig.11 As shown, the image data processing device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the image data processing device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and a keyboard (Keyboard). The user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.11 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.
[0204] exist Fig.11 In the image data processing device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:
[0205] Acquire an original image, and determine a plurality of areas to be selected in the original image;
[0206] Acquire a position region to which each of the multiple regions to be selected belongs in the original image; the position region includes the region to be selected, and the size of the position region is larger than the size of the region to be selected;
[0207] Determine the images in the position area except the area to be selected as surrounding images corresponding to the area to be selected;
[0208] According to the surrounding images corresponding to each of the to-be-selected areas, a target area is determined among the multiple to-be-selected areas; the target area is an area in the original image used for inserting a business image.
[0209] It should be understood that the image data processing device 1000 described in the embodiment of the present invention can execute the above Figure 2 and Figure 3 The description of the image data processing method in any of the corresponding embodiments may also be performed as described above. Fig.10 The description of the image data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.
[0210] In addition, it should be pointed out that: the embodiment of the present invention further provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the image data processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 2 and Figure 3 The description of the image data processing method in any of the corresponding embodiments of the present invention will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here. For technical details not disclosed in the computer storage medium embodiment of the present invention, please refer to the description of the method embodiment of the present invention.
[0211] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0212] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A method for processing image data, characterized in that: include: Acquire an original image, and determine a plurality of areas to be selected in the original image; Obtaining a location area in the original image to which each of the multiple areas to be selected belongs; The location area includes the area to be selected, and the size of the location area is larger than the size of the area to be selected; Determine the images in the position area except the area to be selected as surrounding images corresponding to the area to be selected; Determine a target area in the multiple areas to be selected according to the surrounding images corresponding to each area to be selected; the target area is an area in the original image for inserting a business image; Wherein, determining the target area among the multiple areas to be selected according to the surrounding images corresponding to each area to be selected includes: Extracting surrounding image features from the surrounding image, and determining a service applicability evaluation parameter corresponding to image content contained in the surrounding image according to the surrounding image features; Determine the area type of each area to be selected according to the business applicability evaluation parameter; the area type includes a business area type and a non-business area type; The to-be-selected area whose area type is the service area type is determined as the target area.
2. The method according to claim 1, characterized in that The service applicability evaluation parameter is a service applicability probability; and determining the area type of each area to be selected according to the service applicability evaluation parameter includes: Determine the type of the area to be selected corresponding to the image content where the service applicability probability is greater than or equal to the probability threshold as the service area type; The type of the area to be selected corresponding to the image content whose service applicability probability is less than the probability threshold is determined as the non-service area type.
3. The method according to claim 1, characterized in that The extracting surrounding image features from the surrounding image and determining the service applicability evaluation parameters corresponding to the image content contained in the surrounding image according to the surrounding image features includes: Inputting the surrounding images corresponding to each area to be selected into the input layer of the deep residual network model; In the deep residual network model, extracting surrounding image features in the surrounding image; A convolution operation is performed on the surrounding image features based on the convolution block in the deep residual network model, and a business applicability evaluation parameter corresponding to the image content in the surrounding image is determined based on the convolution operation result.
4. The method according to claim 1, characterized in that: The step of determining the to-be-selected area whose area type is the business area type as the target area includes: In the to-be-selected area whose area type is the business area type, determining the to-be-selected area that meets the area aspect ratio rule as the key selection area; The target area is determined from the key selection areas according to a first quantity threshold of the business images and a business applicability evaluation parameter corresponding to each key selection area.
5. The method according to claim 1, characterized in that The determining of a plurality of areas to be selected in the original image comprises: determining a second quantity threshold for selecting a region and a region size; The to-be-selected regions are randomly determined in the original image according to the second quantity threshold and the region size; the number of the to-be-selected regions is the second quantity threshold.
6. The method according to claim 3, characterized in that Also includes: Acquire a sample fusion image; the sample fusion image includes a sample original image and a sample business image; the sample original image includes position mark information corresponding to the sample business image; Sampling is performed in the sample original image according to the position mark information to obtain a positive sample sampling image and a negative sample sampling image associated with the sample business image, and an initial neural network model is trained based on the positive sample sampling image and the negative sample sampling image; The initial neural network model trained to a converged state is determined as the deep residual network model.
7. The method according to claim 6, characterized in that The step of sampling the sample original image according to the position mark information to obtain a positive sample sample image and a negative sample sample image associated with the sample business image, and training an initial neural network model based on the positive sample sample image and the negative sample sample image includes: Determine, according to the position mark information, a sample position area to which the sample business image belongs in the sample original image; the sample position area includes the sample business image, and a size of the sample position area is larger than a size of the sample business image; Sampling sample surrounding images in images other than the sample business image in the sample position area, and sampling sample far-away images in images other than the sample position area in the original image; Setting a positive sample label for the sample surrounding image, and determining the sample surrounding image with the positive sample label as the positive sample sampling image; Setting a negative sample label for the sample far-away image, and determining the sample far-away image with the negative sample label as the negative sample sampling image; The initial neural network model is trained based on the positive sample sampling images and the negative sample sampling images.
8. The method according to claim 6, characterized in that The step of sampling the sample original image according to the position mark information to obtain a positive sample sample image and a negative sample sample image associated with the sample business image, and training an initial neural network model based on the positive sample sample image and the negative sample sample image includes: Obtaining the number of iterative trainings of the initial neural network model, and determining the training stages of the initial neural network model for the positive sample image and the negative sample image according to the number of iterative trainings; the training stages include a first training stage and a second training stage; A first training phase corresponding to the initial neural network model is performed based on a first learning rate, and a second training phase corresponding to the initial neural network model is performed based on a second learning rate.
9. An image data processing device, characterized in that: include: A first determination module is used to obtain an original image and determine a plurality of areas to be selected in the original image; A first acquisition module is used to acquire a location area to which each of the multiple to-be-selected areas belongs in the original image; The location area includes the area to be selected, and the size of the location area is larger than the size of the area to be selected; A second determining module, used to determine the images in the position area except the area to be selected as surrounding images corresponding to the area to be selected; A third determination module is used to determine a target area in the multiple areas to be selected according to the surrounding images corresponding to each area to be selected; the target area is the area in the original image for inserting the business image; The third determining module determines the target area among the multiple areas to be selected according to the surrounding images corresponding to each area to be selected, including: Extracting surrounding image features from the surrounding image, and determining a service applicability evaluation parameter corresponding to image content contained in the surrounding image according to the surrounding image features; Determine the area type of each area to be selected according to the business applicability evaluation parameter; the area type includes a business area type and a non-business area type; The to-be-selected area whose area type is the service area type is determined as the target area.
10. The device according to claim 9, characterized in that The third determining module comprises: an extraction unit, configured to extract surrounding image features from the surrounding image, and determine a service applicability evaluation parameter corresponding to image content contained in the surrounding image according to the surrounding image features; A first determining unit is used to determine the area type of each area to be selected according to the business applicability evaluation parameter; the area type includes a business area type and a non-business area type; The second determining unit is configured to determine the to-be-selected area whose area type is the service area type as the target area.
11. The device according to claim 10, characterized in that The service applicability evaluation parameter is a service applicability probability; the image data processing device further includes: A first type determination module, configured to determine the type of the area to be selected corresponding to the image content whose service applicability probability is greater than or equal to a probability threshold as the service area type; The second type determination module is used to determine the type of the to-be-selected area corresponding to the image content whose service applicability probability is less than the probability threshold as the non-service area type.
12. The device according to claim 10, characterized in that The extraction unit comprises: An input subunit, used for inputting the surrounding image corresponding to each area to be selected into the input layer of the deep residual network model; An extraction subunit, configured to extract surrounding image features from the surrounding image in the deep residual network model; A determination subunit is used to perform a convolution operation on the surrounding image information based on the convolution block in the deep residual network model, and determine the business applicability evaluation parameters corresponding to the image content in the surrounding image based on the convolution operation result.
13. An image data processing device, characterized in that: include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program to execute the method according to any one of claims 1 to 8.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is executed.
Citation Information
Patent Citations
Method and device for embedding picture
CN106991641A
Image object recognition
CN109146074A
Object detecting device, and imaging apparatus
JP2009194786A
System and method for locating color and pattern match regions in a target image
US6963425B1
