Image design element extraction method, extraction device and electronic equipment

Abstract

The present invention provides a method, device, and electronic device for extracting image design elements. The method for extracting image design elements includes: acquiring a target image; inputting the target image into a target neural network model, and obtaining multiple first element image blocks corresponding to the target image based on the output result of the target neural network model; receiving a first input from a user; and determining a target first element image block from the multiple first element image blocks in response to the first input, wherein the target first element image block is used for subsequent optimization of the target neural network model. The method for extracting image design elements of the present invention can extract first element image blocks with highly semantically abstract features and artistic design sense, with high extraction accuracy and good extraction quality; and the target neural network model is continuously optimized through long-term use and screening by users and interactive online learning, and has strong autonomous learning capabilities and broad application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, an extraction device and an electronic device for extracting image design elements. Background Art

[0002] Image design element extraction refers to the process of extracting design elements from artistic images that can be used in industrial design. Related technologies primarily use methods such as image detection, edge extraction, or neural style transfer to extract design element images. However, image blocks extracted through image detection lack aesthetic appeal and have low design usability. Edge extraction can only extract lines within an image, while neural style transfer methods can only be applied in specific scenarios. None of these methods can extract high-quality design elements that can be used in design. Summary of the Invention

[0003] The present invention provides a method, device and electronic device for extracting image design elements, which screen high-quality design element data through user use and interaction, and continuously iterate and optimize the design element extraction model through online learning, so as to solve the defect of low execution efficiency of image element extraction in the existing technology and realize high-quality image element extraction.

[0004] The present invention provides a method for extracting image design elements, comprising:

[0005] Acquire the target image;

[0006] Inputting the target image into a target neural network model, and obtaining a plurality of first element image blocks corresponding to the target image according to an output result of the target neural network model;

[0007] receiving a first input from a user;

[0008] In response to the first input, determining a target first element image block from the plurality of first element image blocks, the target first element image block being used for subsequent optimization of the target neural network model;

[0009] The target neural network model is obtained by training with a sample image as a sample and a sample first element image block corresponding to the sample image as a sample label.

[0010] According to a method for extracting image design elements provided by the present invention, after determining a target first element image block from the plurality of first element image blocks in response to the first input, the method further includes:

[0011] The target first element image block is stored in a training sample to update the training library of the target neural network model, and the updated training library is used to guide subsequent update optimization of the target neural network model.

[0012] According to a method for extracting image design elements provided by the present invention, the target neural network model includes a detection unit and a screening unit. The method of inputting the target image into the target neural network model and obtaining a plurality of first element image blocks corresponding to the target image according to the output result of the target neural network model includes:

[0013] Inputting the target image into the detection unit, the detection unit extracts features from the target image based on semantic features, and outputs a plurality of second element image blocks corresponding to the target image;

[0014] The plurality of second element image blocks are input to the screening unit, and the screening unit screens the plurality of second element image blocks based on design art features, and outputs a plurality of first element image blocks corresponding to the target image.

[0015] According to a method for extracting image design elements provided by the present invention, the target image is input into the detection unit, the detection unit extracts features from the target image based on semantic features, and outputs a plurality of second element image blocks corresponding to the target image, including:

[0016] Inputting the target image into the scaling layer of the detection unit, scaling the target image to a first target resolution, and generating a first image;

[0017] The first image is input into the first feature extraction layer, the first downsampling and feature extraction block and the first result output block of the detection unit, the first feature extraction layer and the first downsampling and feature extraction block are used to perform feature extraction on the first image, and the first result output block is used to output the multiple second element image blocks, where the second element image blocks include the type of the second element image block and the position coordinates of the second element image block.

[0018] According to a method for extracting image design elements provided by the present invention, the method includes inputting the plurality of second element image blocks into the screening unit, the screening unit screening the plurality of second element image blocks based on design art features, and outputting a plurality of first element image blocks corresponding to the target image, including:

[0019] inputting the plurality of second element image blocks into the expansion and scaling layer of the screening unit, respectively adjusting the plurality of second element image blocks to a second target resolution, and generating a plurality of third element image blocks;

[0020] The multiple third element image blocks are input into the second feature extraction layer, the second downsampling and feature extraction block, and the second result output block of the screening unit, and the third element image block whose confidence probability exceeds the target threshold is determined as the first element image block, and the first element image block is output.

[0021] According to a method for extracting image design elements provided by the present invention, before inputting the target image into the target neural network model, the method further includes:

[0022] Respectively acquiring sample images of multiple image types and a sample first element image block corresponding to each of the sample images;

[0023] A combination of each sample image under each image type and the sample first element image block corresponding to each sample image is used as a training sample to obtain multiple training samples corresponding to each image type, and the target neural network model corresponding to each image type is trained using the multiple training samples.

[0024] The present invention also provides a device for extracting image design elements, comprising:

[0025] A first acquisition module is used to acquire a target image;

[0026] A first generating module is configured to input the target image into a target neural network model, and obtain a plurality of first element image blocks corresponding to the target image according to an output result of the target neural network model;

[0027] A first user interaction module, configured to receive a first input from a user;

[0028] a second generating module, configured to determine, in response to the first input, a target first element image block from the plurality of first element image blocks, wherein the target first element image block is used to optimize the target neural network model;

[0029] The target neural network model is obtained by training with a sample image as a sample and a sample first element image block corresponding to the sample image as a sample label.

[0030] The present invention also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described methods for extracting image design elements are implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for extracting image design elements.

[0032] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of any of the above-mentioned methods for extracting image design elements.

[0033] The image design element extraction method, extraction device and electronic device provided by the present invention perform feature recognition on a target image through a target neural network model to extract a first element image block with highly semantically abstract features and an artistic design sense. The extraction accuracy is high and the quality of the extracted first element image block is good. Moreover, through long-term use and screening by users, first element image blocks that meet the design requirements are added to a training library. The network model is continuously optimized through interactive online learning. The method has a strong autonomous learning ability and can extract high-quality element image blocks in any scenario. The method has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 This is one of the flow charts of the method for extracting image design elements provided by the present invention;

[0036] Figure 2 This is the second flow chart of the method for extracting image design elements provided by the present invention;

[0037] Figure 3 This is the third flow chart of the method for extracting image design elements provided by the present invention;

[0038] Figure 4 This is the fourth flow chart of the method for extracting image design elements provided by the present invention;

[0039] Figure 5 This is the fifth flow chart of the method for extracting image design elements provided by the present invention;

[0040] Figure 6 It is a structural diagram of the image design element extraction system provided by the present invention;

[0041] Figure 7 It is a structural schematic diagram of the device for extracting image design elements provided by the present invention;

[0042] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] The following combination Figures 1 to 5 The method for extracting image design elements of the present invention is described.

[0045] The execution subject of the image design element extraction method may be an image design element extraction device, or a server, or a user's terminal, such as a mobile phone or a computer.

[0046] like Figure 1 As shown, the method for extracting image design elements includes: step 110, step 120, step 130 and step 140.

[0047] Step 110: Acquire a target image;

[0048] In this step, the target image is the image that needs to be extracted.

[0049] Elements are points, lines, or surfaces in the target image. These elements include those with semantic features and those without. Elements with semantic features have specific shapes and meanings, and can be used in subsequent process design, such as flowers, plants, fish, and insects. Elements without semantic features can be basic graphic elements, such as ordinary circles or triangles.

[0050] It should be noted that semantic features can be customized by users.

[0051] Under different definitions, the semantic features may be different.

[0052] The target image may be an image of any style, such as a thangka image or other non-thangka images. It can be understood that any image includes at least one element.

[0053] Taking thangka images as an example, a landscape thangka image may include one or more elements such as flowers, trees, birds, fish, snakes, dragons, elephants, seven treasures, ritual instruments, cloud patterns, fire patterns, water patterns, rocks and fabric patterns.

[0054] In the industrial design process, it is often necessary to extract elements from images to obtain elements with semantic features, and use the extracted image elements in other image designs or process designs.

[0055] In actual execution, the target image may be acquired through an image sensor, downloaded from the Internet, retrieved from a local database, or acquired through other methods, which are not limited in the present invention.

[0056] In some embodiments, after the target image is acquired, the target image can be stored in a local or cloud database and can be called when needed.

[0057] Step 120: Input the target image into the target neural network model, and obtain a plurality of first element image blocks corresponding to the target image according to the output result of the target neural network model;

[0058] Wherein, the target neural network model is obtained by training with the sample image as the sample and the sample first element image block corresponding to the sample image as the sample label;

[0059] In this step, the target neural network model is a network model used to extract element features from the target image.

[0060] Among them, the target neural network model is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing.

[0061] The input data of the target neural network model is the target image, and the output data of the target neural network model is a plurality of first element image blocks, that is, a set of first element image blocks.

[0062] The first element image block is an element image block having a high degree of semantic abstraction and artistic design sense among all elements included in the target image.

[0063] There may be multiple first element image blocks.

[0064] For example, for a landscape thangka image, after being processed by the target neural network model, multiple first-element image blocks such as flowers, fish, snakes, and cloud patterns are finally output.

[0065] It is understandable that before using the target neural network model to extract elements, the target neural network model needs to be trained first. When training the target neural network model, a sample image and a sample first element image block corresponding to the sample image are required.

[0066] The sample images may be images of any type or style, and each sample image corresponds to one or more sample first element image blocks.

[0067] The first element image block of the sample is a marked element having semantic features on the sample image.

[0068] It should be noted that, in some embodiments, the sample first element image block may further include a target first element image block. The method for generating the target first element image block will be specifically described in subsequent embodiments and will not be elaborated here.

[0069] In the actual implementation process, the target neural network model can be trained using the sample image as the sample and the sample first element image block corresponding to the sample image as the sample label. The present invention will specifically describe the implementation method of this step in subsequent embodiments, and will not be repeated here.

[0070] It should be noted that the multiple first element images generated by step 120 can be used for user screening in subsequent processes. The user selects the first element image blocks that meet the design requirements. Based on the first element image blocks that meet the design requirements, the target neural network model can be trained and optimized and updated.

[0071] Step 130: Receive a first input from the user;

[0072] In this embodiment, the first input is used to determine a target first element image block from the plurality of first element image blocks generated in step 120 .

[0073] The target first element image block is a design element image block to be selected by the user. The number of the target first element image blocks may be one or more, which is not limited in the present invention.

[0074] The first input may be in at least one of the following ways:

[0075] First, the first input may be a touch operation, including but not limited to a click operation, a slide operation, and a press operation.

[0076] In this embodiment, receiving the first input from the user may be receiving a touch operation of the user on the display area of ​​the terminal display screen.

[0077] In order to reduce the user error operation rate, the effective area of ​​the first input can be limited to a specific area, such as the upper middle area of ​​the interface displaying multiple first element image blocks; or when the interface displays the first element image block, the target control is displayed on the current interface, and the first input can be achieved by touching the target control; or the first input can be set to click the corresponding first element image block to achieve the first input.

[0078] Secondly, the first input may be a physical key input.

[0079] In this embodiment, the terminal body is provided with corresponding physical buttons, such as a mouse or keyboard, etc., to receive the user's first input, which can be to receive the first input of the user moving or pressing the corresponding physical button; the first input can also be a combination operation of pressing multiple physical buttons at the same time.

[0080] Third, the first input may be voice input.

[0081] In this embodiment, when receiving a voice such as "select moiré", the terminal may perform voice recognition on the voice to generate a control instruction for determining the moiré element image block as the target first element image block.

[0082] Of course, in other embodiments, the first input may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs and is not limited in this embodiment of the present application.

[0083] Among them, users can be designers, etc.

[0084] It should be noted that after multiple first element image blocks are generated in step 120, the multiple first element image blocks can be displayed. For example, the multiple first element image blocks can be displayed on the display screen of the terminal, or can be displayed on a screen connected to the terminal for communication, or can be displayed in other forms, which is not limited by the present invention.

[0085] The displayed multiple first element image blocks are used for the user to review or perform subsequent selection operations, including but not limited to allowing the user to select element image blocks that meet the design requirements for subsequent optimization of the target neural network model.

[0086] Step 140: In response to the first input, determine a target first element image block from multiple first element image blocks, and the target first element image block is used for subsequent optimization of the target neural network model.

[0087] In the actual implementation process, Figure 2 As shown, an interactive use unit may be provided to execute this embodiment, thereby enabling interactive selection by the user.

[0088] like Figure 5 As shown, continuing with the thangka image as an example, by inputting a thangka image into the target neural network model, after processing, it finally outputs multiple first element image blocks such as flowers, trees, birds, fish, snakes, dragons, elephants, seven treasures, ritual instruments, cloud patterns, fire patterns, water patterns, rocks and fabric patterns.

[0089] The terminal displays the multiple first element image blocks.

[0090] In this display interface, the user selects the design element image block to be used by clicking the corresponding first element image block on the screen. For example, the user can click the four first element image blocks of the dragon, elephant, seven treasures and magic tools in the first element image block to achieve the first input.

[0091] In response to the first input, the terminal determines the four first element image blocks of the dragon, elephant, seven treasures and magic tools as target first element image blocks. In some embodiments, the terminal can also output the four design element image blocks selected by the user to a specified path or save them.

[0092] Among them, the target first element image block can be used in the user's subsequent process design, or can also be used for subsequent optimization of the target neural network model.

[0093] According to the method for extracting image design elements provided by an embodiment of the present invention, feature recognition is performed on the target image through a target neural network model to extract a first element image block with highly semantically abstract features and an artistic design sense. The extraction accuracy is high and the quality of the extracted first element image block is good. Moreover, through the results of long-term use and screening by users, the target neural network can be further optimized in turn. The method has a strong autonomous learning ability and can extract high-quality element image blocks in any scenario, and has broad application prospects.

[0094] In some embodiments, after step 140, the method may further include: storing the target first element image block to a training sample to update the training library of the target neural network model, and the updated training library is used to guide subsequent update optimization of the target neural network model.

[0095] In this embodiment, the training samples are samples used to train the target neural network model, and the training samples are stored in the training library. The generation method of the training samples will be described in subsequent embodiments and will not be elaborated here.

[0096] After the terminal determines the target first element image block in response to the user's first input, it can store the target first element image block as a positive sample in the training library to update and expand the training library. In subsequent model optimization training, optimization training based on the updated training library can significantly improve the accuracy of the output results of the target neural network model.

[0097] For example, during the actual execution process, the user clicks on the four first element image blocks of the dragon, the elephant, the seven treasures, and the magic instrument in the first element image block to implement the first input.

[0098] In response to the first input, the terminal determines the four first element image blocks of dragon, elephant, seven treasures and magic tools as target first element image blocks, and outputs the four design element image blocks selected by the user to the training library for storage.

[0099] Of course, in other embodiments, when the displayed multiple first element image blocks do not meet the user's requirements, the terminal can also receive the user's second input. In response to the second input, the terminal deletes the displayed multiple first element image blocks and obtains a new target image to re-execute the above steps 120 to 140.

[0100] The second input may be a touch input, physical key input, voice input, character input, etc., which is the same as the first input, and the present invention does not limit this.

[0101] In this embodiment, the user can select the generated first element image block through the first input to select the design element image block to be used according to actual needs, and add the design element image block that the user believes meets the design requirements to the target neural network model training library. Through long-term use and screening by the user, the first element image block that meets the design requirements is added to the training library, and the neural network model is continuously optimized through interactive online learning, which has high flexibility of use, universality and human-computer interactivity.

[0102] The following describes step 120 in detail through an embodiment.

[0103] In some embodiments, the target neural network model includes a detection unit and a screening unit, and step 120 includes:

[0104] Inputting the target image into the detection unit, the detection unit extracts features of the target image based on semantic features and outputs a plurality of second element image blocks corresponding to the target image;

[0105] The plurality of second element image blocks are input to a screening unit, which screens the plurality of second element image blocks based on design art features and outputs a plurality of first element image blocks corresponding to the target image.

[0106] In this embodiment, if Figure 2 As shown, the target neural network model includes: a detection unit and a screening unit.

[0107] Among them, the detection unit is used to perform a first feature extraction on the target image, detect a set of second image blocks with certain semantics from the target image, and output multiple second element image blocks and position information of each second element image block in the target image.

[0108] The second element image blocks are elements in the target image that have highly semantically abstract features, and the number of the second element image blocks is greater than or equal to the number of the first element image blocks.

[0109] The screening unit is used to perform secondary recognition and screening on the plurality of second element image blocks output by the detection unit, so as to output a first target number of first element image blocks with a sense of design artistry.

[0110] It can be understood that the second element image block is an element image block with a high degree of semantic abstraction; and the first element image block is an element image block with a high degree of semantic abstraction and an artistic sense of design.

[0111] The first target number may be user-defined, for example, it may be set to any value such as 20 or 32.

[0112] The average confidence probability of the first element image block is higher than the average confidence probability of the second element image block.

[0113] Among them, the confidence probability is the probability used to measure the reliability of statistical inference. It can be understood that the higher the confidence probability, the higher the corresponding accuracy.

[0114] For example, in an actual execution process, the target image is an art image X.

[0115] After the art image X is input into the target neural network model, the target neural network model detects the input art image X through the detection unit F detect To obtain a set of second element image blocks with certain semantics, the formula can be used:

[0116] {xi,ci}=F detect (X)

[0117] Get, where {x i , c i}(i=1, 2, ..., N1) represents a set of second element image blocks with certain semantics, x i Characterize the second element image block, c i Represents the category label corresponding to the second element image block, and N1 is the total number of second element image blocks.

[0118] The category label corresponding to the second element image block is used to characterize the type of the second element image, such as the type of the second element image is flower, tree, bird, or cloud pattern.

[0119] In some embodiments, inputting the target image into the detection unit of the target neural network model and outputting a plurality of second element image blocks corresponding to the target image may include:

[0120] Inputting the target image into the scaling layer of the detection unit, scaling the target image to a first target resolution, and generating a first image;

[0121] The first image is input into the first feature extraction layer, the first downsampling and feature extraction block and the first result output block of the detection unit, the first feature extraction layer and the first downsampling and feature extraction block are used to extract features of the first image, and the first result output block is used to output a plurality of second element image blocks, wherein the second element image blocks include the type of the second element image block and the position coordinates of the second element image block.

[0122] For example, Figure 3 As shown, the detection unit may include: a scaling layer, a first feature extraction layer, a first downsampling and feature extraction block, and a first result output block.

[0123] The scaling layer is used to reduce the resolution of the target image to a first target resolution.

[0124] The first target resolution may be user-defined, for example, it may be set to 512×512.

[0125] After the target image is scaled by the scaling layer, a first image having a resolution of the first target resolution can be obtained.

[0126] By reducing the resolution of the target image, the amount of data in the subsequent data processing process can be significantly reduced, thereby improving the speed and efficiency of data processing.

[0127] Especially for thangka images, the initial resolution of a thangka image can reach 3000× resolution. Reducing its resolution can significantly reduce the number of parameters of the target neural network, improve the efficiency of data processing, and help improve the accuracy of the results.

[0128] The first feature extraction layer may include a 3×3 convolutional layer for decomposing each input first image into a feature map of a second target number of channels.

[0129] The second target number can be user-defined, such as set to 64.

[0130] The first downsampling and feature extraction block may include: a 3×3 convolutional layer with a stride of 2 and a residual block. The residual block may include a first 3×3 convolutional layer, a rectified linear unit activation function layer, a second 3×3 convolutional layer, and a residual connection from the residual block input to the second 3×3 convolutional layer.

[0131] The first downsampling and feature extraction block is used to extract features of various feature sizes to improve extraction precision and accuracy.

[0132] It should be noted that the number of the first downsampling and feature extraction blocks can be one or more, for example, can be set to 5 or 6.

[0133] For example, when the number of first downsampling and feature extraction blocks is 6, the first image processed by the first feature extraction layer will pass through 6 first downsampling and feature extraction blocks in sequence, and the final processed data will be input into the first result output block.

[0134] The first result output block may include two output branches.

[0135] The output branch 1 may include a 3×3 convolutional layer, which outputs the second resolution and 4 channels for each first image, corresponding to the detected features Figure 4 The position offset of the corner pixel point is used to determine the position coordinates of each second element image block on the first image.

[0136] The second output branch may include a 3×3 convolution layer, which outputs an output of the second resolution and a channel of 1 for each first image, and is used to represent the predicted category of the target element image block in the small area.

[0137] The value of the second resolution may be determined based on the first target resolution and the second target number.

[0138] For example, after the target image is scaled in the scaling layer to obtain a first image with a resolution of 512×512, the first image is input into the first feature extraction layer, and the first image is decomposed into a 64-channel feature map through feature extraction.

[0139] The feature map is then input into the first downsampling and feature extraction block. After processing, the first branch outputs an output with a resolution of 8×8 and 4 channels, and the second branch outputs an output with a resolution of 8×8 and 1 channel. The type and position coordinates of each second element image block can be obtained.

[0140] It should be noted that the detection unit is used to detect element image blocks with semantic labels in the target image.

[0141] In other embodiments, the structure of the detection unit may be adjusted, such as adjusting the number of convolutional layers or the number of convolution kernels based on actual needs, which is not limited in the present invention.

[0142] Of course, in other embodiments, a commonly used image detection network can also be used in this detection unit through transfer training, and the present invention does not limit this.

[0143] It is understandable that the detection unit needs to be trained in advance. The present invention will describe its training method in detail in the subsequent embodiments.

[0144] In the above embodiment, the target image is processed for the first time by the detection unit to identify and extract the element image block features in the target image to generate a plurality of second element image blocks, and the feature extraction effect is good and the accuracy is high.

[0145] After the second element image blocks are obtained, the plurality of second element image blocks are input into a screening unit to screen the plurality of second element image blocks to obtain a first target number of first element image blocks.

[0146] The step will be described by continuing to take the target image as the art image X as an example.

[0147] After the art image X is processed for the first time through the above steps to obtain a plurality of second element image blocks {xi, ci}, the second element image block set {xi, ci} is input to the screening unit. select , filter out the first element image block set {x from the second element image block set {xi, ci} j , c j}, (j=1, 2, ..., N2), that is, by the formula:

[0148] {x j , c j}=F select ({xi,ci})(j=1,2,…,N2)

[0149] Get multiple second element image blocks, where x j Characterize the first element image block, c j Represents the category label corresponding to the first element image block, N2 represents the number of the first element image blocks after screening, and generally satisfies N2< <N1。

[0150] In some embodiments, inputting the plurality of second element image blocks into a screening unit of a target neural network model and outputting the plurality of first element image blocks corresponding to the target image may include:

[0151] inputting the plurality of second element image blocks into the expansion and scaling layer of the screening unit, respectively adjusting the plurality of second element image blocks to a second target resolution, and generating a plurality of third element image blocks;

[0152] Multiple third element image blocks are input into the second feature extraction layer, the second downsampling and feature extraction block, and the second result output block of the screening unit, and the third element image blocks whose confidence probability exceeds the target threshold are determined as the first element image blocks, and the first element image blocks are output.

[0153] For example, Figure 4 As shown, the screening unit can be composed of a binary classification network, and the screening unit can include: an expansion and scaling layer, a second feature extraction layer, a second downsampling and feature extraction block, and a second result output block.

[0154] The expansion and scaling layer is used to uniformly adjust the second element image blocks in the second element image block set to the second target resolution to generate a third element image block.

[0155] The second target resolution may be user-defined, for example, set to 256×256.

[0156] In some embodiments, if the number of second element image blocks in the second element image block set is less than the third target number, the expansion and scaling layer can also add all-zero image blocks to supplement the second element image set, and fill the number of image blocks in the set to the third target number.

[0157] The third target number can be user-defined, such as set to 128 or other values.

[0158] The second feature extraction layer may include a 3×3 convolutional layer for decomposing each input third element image block into a feature map of a fourth target number of channels.

[0159] The fourth target number can be user-defined, such as set to 64.

[0160] The second downsampling and feature extraction block may include: a pixel shuffling layer, two residual blocks and a shrinking layer.

[0161] The pixel shuffling layer is used to split each 2×2 local four pixel points of the feature map output by the second feature extraction layer into four downsampled feature maps.

[0162] Taking a 256×256 feature map as an example, it can be converted into four 128×128 downsampled feature maps through the pixel shuffling layer.

[0163] Each residual block can include: the first 3×3 convolutional layer, a rectified linear unit activation function layer, the second 3×3 convolutional layer, and a residual connection from the residual block input to the second 3×3 convolutional layer.

[0164] The shrinking layer may include a 1×1 convolution layer for reducing the channels of the feature map whose number of channels is increased after being processed by the pixel shuffling layer to the original number of channels.

[0165] For example, when the pixel shuffling layer expands the number of feature map channels to four times the original number, the shrinking layer reduces the number of feature map channels to one quarter.

[0166] It should be noted that the number of the second downsampling and feature extraction blocks can be one or more, for example, can be set to five or six, and the present invention does not limit this.

[0167] When the number of second downsampling and feature extraction blocks is set to 5, the feature map processed by the second feature extraction layer will pass through 5 second downsampling and feature extraction blocks in sequence, and the final processed data will be input into the second result output block.

[0168] The second result output block can include a 1×1 convolutional layer, two fully connected layers, and a softmax layer. The convolutional layer is used to reduce the number of channels to 16, the fully connected layer is used to expand the feature map into one dimension for operation, and the output dimension is 256; the fully connected layer is used to output the dimension to 128; and the softmax layer is used to output the confidence probability corresponding to the third element image block.

[0169] In a case where the confidence probability corresponding to the third element image block exceeds the target threshold, the third element image block is determined as the first element image block.

[0170] The target threshold is used to select an element image block with a higher confidence probability from a plurality of third element image blocks.

[0171] The target threshold can be user-defined, such as 0.6 or 0.7.

[0172] In some embodiments, when there are a large number of third element image blocks whose confidence probability exceeds the target threshold, it is also possible to sort them based on the confidence probability of all third element image blocks corresponding to the target image, and determine the first target number of third element image blocks with the highest confidence probability as the first element image blocks.

[0173] The first target number can be user-defined, such as 32 or 48.

[0174] The third element image block set output by the screening unit is the final generated design element image block.

[0175] It is understandable that the screening unit also needs to be trained in advance. The present invention will specifically describe its training method in subsequent embodiments.

[0176] In other embodiments, the structure of the screening unit may be adjusted, such as adjusting the number of convolutional layers or the number of convolution kernels based on actual needs, which is not limited in the present invention.

[0177] In the above embodiment, the second element image block generated by the detection unit is screened twice by the screening unit to generate a first element image block with a higher confidence probability, which can further improve the precision and accuracy of the recognition, thereby improving the quality of the element image block finally extracted.

[0178] The following describes the training method of the target neural network model through specific examples.

[0179] In some embodiments, before step 120, the method further includes:

[0180] Respectively acquiring sample images of multiple image types and a sample first element image block corresponding to each sample image;

[0181] The combination of each sample image under each image type and the sample first element image block corresponding to each sample image is used as a training sample to obtain multiple training samples corresponding to each image type, and the target neural network model corresponding to each image type is trained using the multiple training samples.

[0182] In this embodiment, the acquired sample images are first classified to obtain different types of sample images, such as thangka sample images and non-thangka sample images.

[0183] Through classification training, the extraction effect can be improved and the learning difficulty can be reduced.

[0184] Semantic features are defined for each type of sample image respectively, and based on the semantic features, the elements in each sample image that meet the semantic features are marked, so as to obtain the sample first element image block corresponding to each sample image of the same type.

[0185] The combination of each sample image of the same type and the sample first element image block corresponding to the sample image is used as a training sample to obtain multiple training samples of the same type. The target neural network model is trained using multiple training samples to obtain the target neural network model corresponding to the image type.

[0186] It should be noted that, in the actual implementation process, the target first element image features determined by the user during each use can be used as sample data for subsequent updating and optimization of the target neural network model. As the sample data continues to expand, the learning function of the target neural network will become more powerful, the application scenarios will continue to expand, and the accuracy and precision of the final output results will continue to increase.

[0187] In some embodiments, when the target neural network model includes a detection unit and a screening unit, training the target neural network model may include training the detection unit and the screening unit separately.

[0188] For the detection unit, the detection unit can be trained for different types of sample images to improve the extraction effect and reduce the learning difficulty.

[0189] Specifically, sample images of multiple image types and a sample second element image block corresponding to each sample image are respectively obtained;

[0190] The combination of each sample image of each image type and the sample second element image block corresponding to each sample image is used as a training sample to obtain multiple training samples corresponding to each image type, that is, a training set, and the detection unit corresponding to each image type is trained using the training set.

[0191] Taking thangka images as an example, we first collected thangka images to build a thangka image library, and then manually annotated the training set. For example, we categorized and annotated image blocks of commonly used thangka elements, such as flowers, trees, birds, fish, snakes, dragons, elephants, seven treasures, ritual instruments, cloud patterns, fire patterns, water patterns, rocks, and fabric patterns.

[0192] After the pre-training is completed, the parameters of the detection unit may no longer be updated and are used to preliminarily extract the first element image block set.

[0193] Continue to refer Figure 5 ,For the screening unit, it should be explained that its training can ,include a pre-training phase and an online updating phase.

[0194] In the pre-training stage, sample second element image blocks of multiple image types, sample first element image blocks corresponding to each sample second element image block, and negative sample image blocks corresponding to each sample second element image block are respectively obtained;

[0195] A combination of the second element image blocks of each sample of each image type, the first element image blocks of each sample corresponding to each second element image block, and the negative sample image blocks is used as a training sample to obtain multiple training samples corresponding to each image type, and the screening unit corresponding to each image type is trained using the multiple training samples.

[0196] Continuing with the example of Thangka images, we use the established design element sample library for training, which includes positive samples and negative samples.

[0197] Among them, the positive sample library includes element image blocks selected by users or designers from sample images for design, and the negative sample library includes image blocks randomly cut from sample images and some image blocks that are not suitable for design and selected from the sample second element image block set.

[0198] After pre-training, the screening unit is available for testing and will be updated online during the use phase.

[0199] During the online update phase, the target element image blocks finally determined by the user during actual use can be used as samples for subsequent training and updated to the positive sample library of the design element sample library, while the element image blocks not selected by the user can be updated to the negative sample library.

[0200] For example, the target first element image block generated by step 140 in the above embodiment can be stored as a positive sample in the training sample, and the first element image block not selected by the user can be stored as a negative sample in the training sample, so as to update and expand the training sample in real time online, thereby realizing the online learning function of the target neural network model.

[0201] In some embodiments, a first number of negative samples may be randomly saved based on a target ratio of positive samples to all samples to avoid an excessive imbalance in the ratio of positive and negative samples.

[0202] The target ratio can be user-defined. Generally, the best effect is achieved when the ratio of positive and negative samples is 1:1.

[0203] The updated and saved positive and negative sample file names and class labels are updated to the data log file.

[0204] It should be noted that, in some embodiments, the method may further include: performing online updates every target period, wherein the target period is used to represent the actual usage time of the user.

[0205] The target period can be customized by the user, for example, half a month or one month. For example, when the user has used the target neural network model for one month, an online update is performed, and the target element image blocks determined by the user within this one month are used as positive samples, and the unselected element image blocks are used as negative samples. These are updated to the design element sample library and used as sample data for the next training. They are then used to train the screening unit and update the parameters of the screening unit. At the same time, the current model is backed up and the model log file is updated.

[0206] Of course, in some embodiments, the method may further include: performing an online update when the number of positive samples reaches a preset value, wherein the preset value can be set to any value such as 5000 or 6000, and the present invention does not limit this.

[0207] The method for extracting image design elements provided by an embodiment of the present invention has strong autonomous learning ability by using the data generated by users during actual use as sample data for the next model training. As the volume of sample data increases, the intelligence level of the model will become higher and higher, the output results will be more accurate, and the scope of application will be wider.

[0208] like Figure 6As shown, in actual implementation, an image element extraction system may be used to execute any step in the above method embodiment. The image element extraction system may include: a data input unit, a data output unit, a storage unit, and a processor.

[0209] The processor is electrically connected to the data input unit, the data output unit and the storage unit respectively, and the storage unit is electrically connected to the data input unit and the data output unit respectively, wherein the electrical connection includes a wired electrical connection and a wireless connection.

[0210] Specifically, a storage unit may be used to store data, and the storage unit includes: storing data generated in each step and storing an executable program, etc. The executable program is used to control the execution of any step of the above method.

[0211] The executable program may be executed by a processor.

[0212] The data input unit may be used to input data, including but not limited to inputting data required for any step.

[0213] A data output unit may be used to output data, including but not limited to outputting data generated by any step.

[0214] The image design element extraction device provided by the present invention is described below. The image design element extraction device described below and the image design element extraction method described above can refer to each other.

[0215] like Figure 7 As shown, the image design element extraction device includes: a first acquisition module 710, a first generation module 720, a first user interaction module 730 and a second generation module 740.

[0216] A first acquisition module 710 is used to acquire a target image;

[0217] A first generating module 720 is configured to input a target image into a target neural network model and obtain a plurality of first element image blocks corresponding to the target image according to an output result of the target neural network model;

[0218] A first user interaction module 730 is configured to receive a first input from a user;

[0219] A second generating module 740 is configured to determine a target first element image block from the plurality of first element image blocks in response to the first input, wherein the target first element image block is used to optimize the target neural network model;

[0220] The target neural network model is obtained by training with the sample image as the sample and the sample first element image block corresponding to the sample image as the sample label.

[0221] According to the image design element extraction device provided by the embodiment of the present invention, feature recognition is performed on the target image through the target neural network model to extract the first element image block with highly semantic abstract features and artistic design sense. The extraction accuracy is high and the quality of the extracted first element image block is good. Moreover, through long-term use and screening by users, the first element image block that meets the design requirements is added to the training library, and the network model is continuously optimized through interactive online learning. It has strong autonomous learning ability, can extract high-quality element image blocks in any scenario, and has broad application prospects.

[0222] In some embodiments, the second generating module 740 is further configured to:

[0223] After determining a target first element image block from a plurality of first element image blocks in response to a first input, the target first element image block is stored in a training sample to update a training library of a target neural network model, and the updated training library is used to guide subsequent update optimization of the target neural network model.

[0224] In some embodiments, the target neural network model includes a detection unit and a screening unit, and the first generation module 720 is further configured to:

[0225] Inputting the target image into the detection unit, causing the detection unit to extract features of the target image based on semantic features and output a plurality of second element image blocks corresponding to the target image;

[0226] The plurality of second element image blocks are input to a screening unit, so that the screening unit screens the plurality of second element image blocks based on the design art features and outputs a plurality of first element image blocks corresponding to the target image.

[0227] In some embodiments, the first generating module 720 is further configured to:

[0228] Inputting the target image into the scaling layer of the detection unit, scaling the target image to a first target resolution, and generating a first image;

[0229] The first image is input into the first feature extraction layer, the first downsampling and feature extraction block and the first result output block of the detection unit, the first feature extraction layer and the first downsampling and feature extraction block are used to extract features of the first image, and the first result output block is used to output a plurality of second element image blocks, wherein the second element image blocks include the type of the second element image block and the position coordinates of the second element image block.

[0230] In some embodiments, the first generating module 720 is further configured to:

[0231] Inputting the plurality of second element image blocks into the expansion and scaling layer of the screening unit, scaling the plurality of second element image blocks to a second target resolution, and generating a plurality of third element image blocks;

[0232] Multiple third element image blocks are input into the second feature extraction layer, the second downsampling and feature extraction block, and the second result output block of the screening unit, and the third element image blocks whose confidence probability exceeds the target threshold are determined as the first element image blocks, and the first element image blocks are output.

[0233] In some embodiments, the apparatus further comprises:

[0234] A second acquisition module is used to respectively acquire sample images of multiple image types and a sample first element image block corresponding to each sample image;

[0235] The second generation module is used to take the combination of each sample image of each image type and the sample first element image block corresponding to each sample image as a training sample, obtain multiple training samples corresponding to each image type, and use the multiple training samples to train the target neural network model corresponding to each image type.

[0236] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a method for extracting image design elements, the method comprising: acquiring a target image; inputting the target image into a target neural network model, and obtaining a plurality of first element image blocks corresponding to the target image according to the output result of the target neural network model; receiving a first input from a user; and determining a target first element image block from the plurality of first element image blocks in response to the first input, the target first element image block being used for subsequent optimization of the target neural network model; wherein the target neural network model is trained using a sample image as a sample and a sample first element image block corresponding to the sample image as a sample label.

[0237] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0238] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image design element extraction method provided by the above methods, and the method includes: obtaining a target image; inputting the target image into a target neural network model, and obtaining multiple first element image blocks corresponding to the target image according to the output result of the target neural network model; receiving a first input from a user; and determining a target first element image block from the multiple first element image blocks in response to the first input, and the target first element image block is used for subsequent optimization of the target neural network model; wherein the target neural network model is trained with a sample image as a sample and a sample first element image block corresponding to the sample image as a sample label.

[0239] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned image design element extraction method provided, the method comprising: obtaining a target image; inputting the target image into a target neural network model, and obtaining a plurality of first element image blocks corresponding to the target image based on the output result of the target neural network model; receiving a first input from a user; and determining a target first element image block from the plurality of first element image blocks in response to the first input, the target first element image block being used for subsequent optimization of the target neural network model; wherein the target neural network model is trained using a sample image as a sample and a sample first element image block corresponding to the sample image as a sample label.

[0240] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0241] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for extracting image design elements, characterized in that: include: Acquire the target image; Inputting the target image into a target neural network model, and obtaining a plurality of first element image blocks corresponding to the target image according to an output result of the target neural network model; The target neural network model includes a detection unit and a screening unit. Inputting the target image into the target neural network model and obtaining a plurality of first element image blocks corresponding to the target image according to an output result of the target neural network model include: Inputting the target image into the detection unit, the detection unit extracts features from the target image based on semantic features, and outputs a plurality of second element image blocks corresponding to the target image; inputting the plurality of second element image blocks into the screening unit, wherein the screening unit screens the plurality of second element image blocks based on design art features and outputs a plurality of first element image blocks corresponding to the target image; receiving a first input from a user; In response to the first input, determining a target first element image block from the plurality of first element image blocks, the target first element image block being used for subsequent optimization of the target neural network model; The target neural network model is obtained by training with a sample image as a sample and a sample first element image block corresponding to the sample image as a sample label.

2. The method for extracting image design elements according to claim 1, characterized in that: After determining a target first element image block from the plurality of first element image blocks in response to the first input, the method further includes: The target first element image block is stored in a training sample to update the training library of the target neural network model, and the updated training library is used to guide subsequent update optimization of the target neural network model.

3. The method for extracting image design elements according to claim 1, wherein: Inputting the target image into the detection unit, wherein the detection unit extracts features from the target image based on semantic features and outputs a plurality of second element image blocks corresponding to the target image, comprises: Inputting the target image into the scaling layer of the detection unit, scaling the target image to a first target resolution, and generating a first image; The first image is input into the first feature extraction layer, the first downsampling and feature extraction block and the first result output block of the detection unit, the first feature extraction layer and the first downsampling and feature extraction block are used to perform feature extraction on the first image, and the first result output block is used to output the multiple second element image blocks, where the second element image blocks include the type of the second element image block and the position coordinates of the second element image block.

4. The method for extracting image design elements according to claim 1, wherein: Inputting the plurality of second element image blocks into the screening unit, wherein the screening unit screens the plurality of second element image blocks based on design art features and outputs a plurality of first element image blocks corresponding to the target image, comprises: inputting the plurality of second element image blocks into the expansion and scaling layer of the screening unit, respectively adjusting the plurality of second element image blocks to a second target resolution, and generating a plurality of third element image blocks; The multiple third element image blocks are input into the second feature extraction layer, the second downsampling and feature extraction block, and the second result output block of the screening unit, and the third element image block whose confidence probability exceeds the target threshold is determined as the first element image block, and the first element image block is output.

5. The method for extracting image design elements according to any one of claims 1 to 4, characterized in that: Before inputting the target image into the target neural network model, the method further includes: Respectively acquiring sample images of multiple image types and a sample first element image block corresponding to each of the sample images; A combination of each sample image under each image type and the sample first element image block corresponding to each sample image is used as a training sample to obtain multiple training samples corresponding to each image type, and the target neural network model corresponding to each image type is trained using the multiple training samples.

6. A device for extracting image design elements, characterized in that: include: A first acquisition module is used to acquire a target image; A first generating module is configured to input the target image into a target neural network model, and obtain a plurality of first element image blocks corresponding to the target image based on an output result of the target neural network model; the target neural network model includes a detection unit and a screening unit, and inputting the target image into the target neural network model and obtaining a plurality of first element image blocks corresponding to the target image based on an output result of the target neural network model comprises: inputting the target image into the detection unit, the detection unit performing feature extraction on the target image based on semantic features, and outputting a plurality of second element image blocks corresponding to the target image; inputting the plurality of second element image blocks into the screening unit, the screening unit screening the plurality of second element image blocks based on design art features, and outputting a plurality of first element image blocks corresponding to the target image; A first user interaction module, configured to receive a first input from a user; a second generating module, configured to determine, in response to the first input, a target first element image block from the plurality of first element image blocks, wherein the target first element image block is used for subsequent optimization of the target neural network model; The target neural network model is obtained by training with a sample image as a sample and a sample first element image block corresponding to the sample image as a sample label.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for extracting image design elements according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for extracting image design elements according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for extracting image design elements according to any one of claims 1 to 5 are implemented.

Images