Model training method, device, electronic device and storage medium

Abstract

The embodiment of the present invention is applicable to the field of artificial intelligence technology, and provides a model training method, device, electronic device and storage medium, wherein the model training method includes: inputting a training image sample into a first model to obtain a first feature of the training image sample; the training image sample represents a page design drawing containing border information; the first feature represents a frame style feature of the border information in the training image sample; inputting a frame component sample into a second model to obtain a second feature of the frame component sample; the frame component sample is tensor data obtained based on the conversion of the border information in the training image sample; the second feature represents the frame style feature corresponding to the frame component sample; based on the similarity between the first feature and the second feature, determining a loss value corresponding to a set loss function; and updating the weight parameters of the first model and / or the second model according to the loss value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method, device, electronic equipment and storage medium. Background Art

[0002] When performing image retrieval, the related technology mainly searches for specific objects in the image, such as people or objects in the image. When learning the frame style retrieval of the image based on the idea of ​​image retrieval, it is easy to learn the information of the specific object in the image, resulting in information redundancy, so that the model has a low retrieval accuracy when performing frame style retrieval of the image. Summary of the invention

[0003] In order to solve the above problems, the embodiments of the present invention provide a model training method, device, electronic device and storage medium to at least solve the problem that the related technology is easy to learn the information of specific objects in the image when performing retrieval learning of the frame style of the image.

[0004] The technical solution of the present invention is achieved in this way:

[0005] In a first aspect, an embodiment of the present invention provides a model training method, the method comprising:

[0006] Input the training image sample into the first model to obtain the first feature of the training image sample; the training image sample represents the page design image containing border information; the first feature represents the frame style feature of the border information in the training image sample;

[0007] Inputting the frame component sample into the second model to obtain a second feature of the frame component sample; the frame component sample is tensor data obtained by converting the border information in the training image sample; the second feature represents the frame style feature corresponding to the frame component sample;

[0008] Determining a loss value corresponding to a set loss function based on a similarity between the first feature and the second feature;

[0009] Update weight parameters of the first model and / or the second model according to the loss value.

[0010] In the above solution, the first model and the second model share an output layer.

[0011] In the above scheme, the first model and the second model are constructed based on the YOLO model.

[0012] In a second aspect, an embodiment of the present invention provides a retrieval method, the method comprising:

[0013] Inputting a search object into a feature extraction model corresponding to the type of the search object to obtain a frame style feature corresponding to the search object; the feature extraction model includes a first model or a second model trained based on a model training method;

[0014] The frame style feature is input into the first database for matching to obtain the search result corresponding to the search object; wherein,

[0015] The types of the search objects include pictures or framework components; the first database is a database related to the types of search results; the types of the search results include pictures or framework components.

[0016] In the above scheme, when the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; inputting the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0017] Determine the similarity between the frame style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the frame style feature;

[0018] The search result is determined based on the similarity.

[0019] In the above scheme, when the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; inputting the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0020] Determine the similarity between the frame style feature and each second feature stored in the first database; the first database stores at least two frame components and the second feature corresponding to each of the at least two frame components; the second feature represents the frame style feature;

[0021] The search result is determined based on the similarity.

[0022] In the above scheme, when the search object is a frame component, the feature extraction model corresponding to the type of the search object is the second model; the step of inputting the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0023] Determine the similarity between the frame style feature and each second feature stored in a first database; the first database stores at least two frame components and the second feature corresponding to each of the at least two frame components; the second feature represents the frame style feature;

[0024] The search result is determined based on the similarity.

[0025] In the above scheme, when the search object is a frame component, the feature extraction model corresponding to the type of the search object is the second model; the step of inputting the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0026] Determine the similarity between the frame style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the frame style feature;

[0027] The search result is determined based on the similarity.

[0028] In a third aspect, an embodiment of the present invention provides a model training device, the device comprising:

[0029] A first input module is used to input a training image sample into a first model to obtain a first feature of the training image sample; the training image sample represents a page design image containing border information; the first feature represents a frame style feature of the border information in the training image sample;

[0030] A second input module is used to input the frame component sample into a second model to obtain a second feature of the frame component sample; the frame component sample is tensor data obtained by converting the border information in the training image sample; the second feature represents the frame style feature corresponding to the frame component sample;

[0031] A determination module, configured to determine a loss value corresponding to a set loss function based on a similarity between the first feature and the second feature;

[0032] An updating module is used to update the weight parameters of the first model and / or the second model according to the loss value.

[0033] In a fourth aspect, an embodiment of the present invention provides a retrieval device, the device comprising:

[0034] A third input module is used to input the search object into a feature extraction model corresponding to the type of the search object to obtain a frame style feature corresponding to the search object; the feature extraction model includes a first model or a second model trained based on a model training method;

[0035] The fourth input module is used to input the frame style feature into the first database for matching, and obtain the search result corresponding to the search object; wherein,

[0036] The types of the search objects include pictures or framework components; the first database is a database related to the types of search results; the types of the search results include pictures or framework components.

[0037] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the steps of the model training method provided in the first aspect or the steps of the retrieval method provided in the second aspect of the embodiment of the present invention.

[0038] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, including: the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the model training method provided in the first aspect of the embodiment of the present invention or the steps of the retrieval method provided in the second aspect are implemented.

[0039] The embodiment of the present invention obtains the first feature of the training image sample by inputting the training image sample into the first model; and obtains the second feature of the framework component sample by inputting the framework component sample into the second model. Based on the similarity between the first feature and the second feature, the loss value corresponding to the set loss function is determined, and the weight parameters of the first model and / or the second model are updated according to the loss value. Among them, the training image sample represents a page design drawing containing border information, and the first feature represents the frame style feature of the border information in the training image sample; the framework component sample is tensor data obtained based on the conversion of the border information in the training image sample, and the second feature represents the frame style feature corresponding to the framework component sample. The embodiment of the present invention determines the loss value corresponding to the set loss function through the similarity between the first feature and the second feature, and updates the weight parameters of the first model and / or the second model according to the loss value, thereby improving the accuracy of the first model and the second model in framework style retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of an implementation flow of a model training method provided by an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of an implementation flow of a retrieval method provided by an embodiment of the present invention;

[0042] Figure 3 It is a schematic diagram of the implementation flow of another retrieval method provided by an embodiment of the present invention;

[0043] Figure 4 It is a schematic diagram of the implementation flow of another retrieval method provided by an embodiment of the present invention;

[0044] Figure 5 It is a schematic diagram of the implementation flow of another retrieval method provided by an embodiment of the present invention;

[0045] Figure 6 It is a schematic diagram of the implementation flow of another retrieval method provided by an embodiment of the present invention;

[0046] Figure 7 It is a schematic diagram of a model training process provided by an application embodiment of the present invention;

[0047] Figure 8 It is a schematic diagram of a search process provided by an application embodiment of the present invention;

[0048] Fig. 9 is a schematic diagram of a model training device provided by an embodiment of the present invention;

[0049] Fig.10 is a schematic diagram of a retrieval device provided by an embodiment of the present invention;

[0050] Fig.11 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] When performing frame-style retrieval learning of images, related technologies tend to learn content information in the image, such as information about people, objects, and buildings in the image. This information is useless for frame-style retrieval learning and is prone to information redundancy, resulting in low retrieval accuracy when the model performs frame-style retrieval. Here, the frame style of an image refers to the layout of elements in the image. The elements in the image include pictures, text, and buttons, etc. The layout and structural relationship of these elements are called the frame style of the image.

[0053] Related technology: The autoencoder-based method can learn the features of the image frame itself, but cannot learn the representation features of the frame semantics. Frame semantics refers to the type of elements placed in each bounding box in the image, including pictures, texts, buttons, etc.

[0054] In view of the shortcomings of the above-mentioned related technologies, an embodiment of the present invention provides a model training method, which can at least improve the accuracy of the model in retrieving frame styles. In order to illustrate the technical solution of the present invention, a specific embodiment is used for illustration below.

[0055] Figure 1 1 is a schematic diagram of an implementation flow of a model training method provided by an embodiment of the present invention. The execution subject of the model training method may be an electronic device, and the electronic device may be a desktop computer, a laptop computer, a server, etc. Figure 1 , model training methods include:

[0056] S101, input a training image sample into a first model to obtain a first feature of the training image sample; the training image sample contains a page design diagram of border information; the first feature represents a frame style feature of the border information in the training image sample.

[0057] Here, the training image sample is a page design image containing border information, such as a front-end design page image of an application software. The border information is a label of the training image sample, and the elements in the training image sample are marked with a bounding box. The border information includes information such as the size of the bounding box and the type of the element in the bounding box.

[0058] The first model may be a yolo-v3 model, a yolo (you only look once) model is a target detection model, and yolo-v3 is the third version of the yolo series of target detection algorithms. In an embodiment of the present invention, the yolo-v3 model is used to predict the position, size, and type of elements in an image using a bounding box.

[0059] The training image sample is input into the first model to obtain the first feature of the training image sample, and the first feature represents the frame style feature of the border information in the training image sample. Here, the first model encodes the image itself, and the first feature is a feature at the image level.

[0060] S102, input the framework component sample into the second model to obtain the second feature of the framework component sample; the framework component sample is tensor data obtained based on the conversion of the border information in the training image sample; the second feature represents the framework style feature corresponding to the framework component sample.

[0061] Here, the frame component is composed of elements such as pictures, texts, and buttons, and one frame component corresponds to one frame style.

[0062] Here, the first model and the second model need to be trained at the same time, and the training data input into the second model is based on the tensor data obtained by converting the border information in the training image samples, that is, the training image samples are used as the training data for the first model, and the tensor data obtained by converting the border information in the training image samples is used as the training data for the second model.

[0063] The second model extracts features from the tensor data to obtain a second feature, which represents the frame style feature corresponding to the frame component sample. The first feature and the second feature both represent frame style features, the first feature is at the image level, and the second feature is at the data level.

[0064] The second model can be a deep neural network (DNN) model or a simplified yolo-v3 model. Simplification refers to reducing the parameters and network layers of the yolo-v3 model. Since the structure of tensor data is relatively simple, there is no need to use a complex yolo-v3 model for feature extraction, so the yolo-v3 model is simplified.

[0065] In one embodiment, the first model and the second model are both constructed based on the yolo model. For example, the first model uses the original yolo-v3 model, and the second model uses the simplified yolo-v3 model.

[0066] It should be understood that the above steps S101 and S102 are performed simultaneously.

[0067] S103: Determine a loss value corresponding to a set loss function based on the similarity between the first feature and the second feature.

[0068] Here, the set loss function is a matching loss function. The purpose of the set loss function is to make the first feature output by the first model and the second feature output by the second model as similar as possible, so that during subsequent retrieval, the image and the framework components can perform feature matching with each other.

[0069] Here, the similarity between the first feature and the second feature can be determined by calculating the Euclidean distance between the first feature and the second feature. The smaller the Euclidean distance, the greater the similarity between the first feature and the second feature. The similarity between the first feature and the second feature can also be determined by calculating the cosine similarity between the first feature and the second feature. The cosine similarity uses the cosine value of the angle between two vectors in the vector space as a measure of the difference between two individuals. Compared with the distance metric, the cosine similarity pays more attention to the difference in direction between the two vectors rather than the distance or length. In contrast to the Euclidean distance, the smaller the value of the cosine similarity, the smaller the similarity between the first feature and the second feature, and the greater the difference.

[0070] After calculating the similarity between the first feature and the second feature, the loss value corresponding to the set loss function is calculated by the similarity. In an embodiment of the present invention, the greater the similarity, the smaller the loss value corresponding to the set loss function; the smaller the similarity, the greater the loss value corresponding to the set loss function. Substitute the similarity into the set loss function to solve the loss value. When the loss value is the smallest, it can be considered that the model training is completed and the similarity between the first feature and the second feature is maximized. In practical applications, the set loss function can be an L2 loss function, which is also called the minimum square error. The minimum value of the set loss function can be obtained by gradient descent.

[0071] S104: Update weight parameters of the first model and / or the second model according to the loss value.

[0072] Model training is an iterative training process. After each loss value is calculated, the weight parameters of the first model and / or the second model are adjusted according to the loss value. Here, the weight parameters are the configuration variables inside the model. Whether a model is applicable or how effective it is depends largely on the setting of the weight parameters. By tuning the weight parameters, the model can be optimized and the model performance can be improved. In practical applications, the update of weight parameters includes increasing and decreasing parameter values.

[0073] After updating the weight parameters of the first model and / or the second model, the training data is used to continue training the first model and the second model after the weight parameters are updated until the set loss function converges. In addition, the first model and the second model need to correspond to a detection loss function respectively, and the model parameters of the first model and the second model are adjusted according to the detection loss function. In other words, the training of the first model and the second model is completed only when the set loss function and the detection loss function converge.

[0074] In one embodiment, the first model and the second model share an output layer. The first model and the second model share an output layer in order to align the output two frame style feature representations.

[0075] In practical applications, the mean reciprocal rank (MRR) indicator can be used as an evaluation indicator to evaluate the training effect of the model. MRR is an internationally common mechanism for evaluating search algorithms, that is, the first result matches, the score is 1, the second match score is 0.5, the nth match score is 1 / n, and if there is no match, the score is 0. The final score is the sum of all scores. The indicators are calculated separately for the output features of the three scales of the yolo-v3 model. Small-scale features can obtain overall structural information. The larger the scale, the more refined the corresponding feature representation.

[0076] The embodiment of the present invention obtains the first feature of the training image sample by inputting the training image sample into the first model; and obtains the second feature of the framework component sample by inputting the framework component sample into the second model. Based on the similarity between the first feature and the second feature, the loss value corresponding to the set loss function is determined, and the weight parameters of the first model and / or the second model are updated according to the loss value. Among them, the training image sample represents a page design drawing containing border information, and the first feature represents the frame style feature of the border information in the training image sample; the framework component sample is tensor data obtained based on the conversion of the border information in the training image sample, and the second feature represents the frame style feature corresponding to the framework component sample. The embodiment of the present invention determines the loss value corresponding to the set loss function through the similarity between the first feature and the second feature, and updates the weight parameters of the first model and / or the second model according to the loss value, thereby improving the accuracy of the first model and the second model in framework style retrieval.

[0077] Figure 2 1 is a schematic diagram of an implementation flow of a search method provided by an embodiment of the present invention. The execution subject of the search method may be an electronic device, and the electronic device may be a desktop computer, a notebook computer, a server, etc. Figure 2 , the search methods include:

[0078] S201, inputting the search object into a feature extraction model corresponding to the type of the search object to obtain a framework style feature corresponding to the search object; the feature extraction model includes a first model or a second model trained based on a model training method.

[0079] Here, the feature extraction model includes a first model and a second model trained using the above-mentioned model training method. Depending on the type of the retrieval object, the retrieval object needs to be input into different feature extraction models. The type of the retrieval object includes a picture or a framework component. If the type of the retrieval object is a picture, the picture is input into the first model. If the type of the retrieval object is a framework component, the framework component is input into the second model.

[0080] If the search object is a picture, the first feature of the picture is extracted through the first model. If the search object is a frame component, the second feature of the frame component is extracted through the second model. Both the first feature and the second feature are frame style features.

[0081] S202, input the frame style features into the first database for matching, and obtain the search results corresponding to the search object; wherein the types of the search object include pictures or frame components; the first database is a database related to the types of search results; the types of the search results include pictures or frame components.

[0082] After the first model and the second model are trained, a first database is created using the first model and the second model. The first database is related to the type of the search result, and the type of the search result includes a picture or a framework component.

[0083] If the type of the retrieval result is a picture, at least two pictures and a first feature corresponding to each of the at least two pictures are stored in the first database, the first feature corresponding to each of the at least two pictures is extracted in advance using the first model, and each of the at least two pictures and the corresponding first feature are stored in the first database.

[0084] If the type of the search result is a framework component, at least two framework components and the second feature corresponding to each of the at least two framework components are stored in the first database, the second model is used to extract the second feature corresponding to each of the at least two framework components in advance, and each of the at least two framework components and the corresponding second feature are stored in the first database. In practical applications, front-end components are generally stored in a front-end component database, and the second model can be used to extract features from all front-end components in the front-end component database, and the features are correspondingly stored in the front-end component database.

[0085] In practical applications, vector search tools such as Faiss or Vearch can be used to construct a first database for search.

[0086] refer to Figure 3 In one embodiment, when the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; the inputting of the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0087] S301, determining the similarity between the framework style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the framework style feature.

[0088] In the case of retrieving pictures based on pictures, the frame style features (first features) of the pictures are first extracted using the first model, the frame style features are matched with the first features stored in the first database, and the similarity between the frame style features and each first feature stored in the first database is calculated. The greater the similarity, the more similar the frame styles corresponding to the two pictures are. Here, the similarity between the frame style features and each first feature stored in the first database can be calculated using the Euclidean distance.

[0089] S302: Determine the search result based on the similarity.

[0090] The above steps calculate the similarity between the frame style feature and each first feature stored in the first database, select the first features whose similarity to the frame style feature is greater than a set value, and use the pictures corresponding to the first features in the first database whose similarity to the frame style feature is greater than the set value as the search results. Here, according to the number of pictures in the pre-set search results, a set number of pictures are obtained from the first database as the search results.

[0091] In practical applications, the k nearest neighbor algorithm can be used to determine the k first features in the first database that are most similar to the frame style feature, and the pictures corresponding to the top-ranked first features among the k first features are used as retrieval results.

[0092] refer to Figure 4 In one embodiment, when the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; the inputting of the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0093] S401, determining the similarity between the framework style feature and each second feature stored in the first database; the first database stores at least two framework components and the second feature corresponding to each of the at least two framework components; the second feature represents the framework style feature.

[0094] In the case of retrieving a frame component based on an image, the frame style feature (first feature) of the image is first extracted using the first model, the frame style feature is matched with the second feature stored in the first database, and the similarity between the frame style feature and each second feature stored in the first database is calculated. The greater the similarity, the more similar the frame style of the image and the frame component is. Here, the similarity between the frame style feature and each second feature stored in the first database can be calculated using the Euclidean distance.

[0095] S402: Determine the search result based on the similarity.

[0096] The above steps calculate the similarity between the frame style feature and each second feature stored in the first database, select the second features whose similarity is greater than a set value, and use the frame components corresponding to the second features in the first database whose similarity to the frame style feature is greater than the set value as the search results. Here, according to the number of frame components in the pre-set search results, a set number of frame components are obtained from the first database as the search results.

[0097] In practical applications, the k-nearest neighbor algorithm can be used to determine the k second features in the first database that are most similar to the frame style features, and the frame components corresponding to the top-ranked second features among the k second features are used as retrieval results.

[0098] refer to Figure 5 In one embodiment, when the search object is a frame component, the feature extraction model corresponding to the type of the search object is the second model; the inputting of the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0099] S501, determining the similarity between the framework style feature and each second feature stored in a first database; the first database stores at least two framework components and the second feature corresponding to each of the at least two framework components; the second feature represents the framework style feature.

[0100] In the case of retrieving a frame component according to the frame component, the frame style feature (second feature) of the frame component is first extracted using the second model. Usually, the user inputs the style information of the frame component, such as the position, size and element category of a specific element in the frame component. The style information of the frame component input by the user is first converted into tensor data, and the tensor data is input into the second model to obtain the frame style feature.

[0101] The frame style feature is matched with the second feature stored in the first database, and the similarity between the frame style feature and each second feature stored in the first database is calculated. The greater the similarity, the more similar the frame style of the picture and the frame component is. Here, the similarity between the frame style feature and each second feature stored in the first database can be calculated by Euclidean distance.

[0102] S502: Determine the search result based on the similarity.

[0103] The above steps calculate the similarity between the frame style feature and each second feature stored in the first database, select the second features whose similarity is greater than a set value, and use the frame components corresponding to the second features in the first database whose similarity to the frame style feature is greater than the set value as the search results. Here, according to the number of frame components in the pre-set search results, a set number of frame components are obtained from the first database as the search results.

[0104] In practical applications, the k-nearest neighbor algorithm can be used to determine the k second features in the first database that are most similar to the frame style features, and the frame components corresponding to the top-ranked second features among the k second features are used as retrieval results.

[0105] refer to Figure 6 In one embodiment, when the search object is a frame component, the feature extraction model corresponding to the type of the search object is the second model; the inputting of the frame style feature into the first database for matching to obtain the search result corresponding to the search object includes:

[0106] S601, determining the similarity between the framework style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the framework style feature.

[0107] When retrieving images based on a frame component, the second model is first used to extract the frame style features (second features) of the frame component. Usually, the user inputs the style information of the frame component, such as the position, size, and element category of a specific element in the frame component. The style information of the frame component input by the user is first converted into tensor data, and the tensor data is input into the second model to obtain the frame style features.

[0108] Then, the framework style feature is matched with the first feature stored in the first database, and the similarity between the framework style feature and each first feature stored in the first database is calculated. Here, the similarity between the framework style feature and each first feature stored in the first database can be calculated by Euclidean distance.

[0109] S602: Determine the search result based on the similarity.

[0110] The above steps calculate the similarity between the frame style feature and each first feature stored in the first database, select the first features whose similarity is greater than a set value, and use the pictures corresponding to the first features in the first database whose similarity to the frame style feature is greater than the set value as the search results. Here, according to the number of pictures in the pre-set search results, a set number of pictures are obtained from the first database as the search results.

[0111] In practical applications, the k nearest neighbor algorithm can be used to determine the k first features in the first database that are most similar to the framework style features, and the framework components corresponding to the top-ranked first features among the k first features are used as retrieval results.

[0112] It can be seen that the above four embodiments provide four application scenarios of the first model and the second model respectively, which can respectively realize image retrieval of images with similar frame styles, image retrieval of frame components with similar frame styles, frame component retrieval of images with similar frame styles, and frame component retrieval of frame components with similar frame styles. Since the first feature output by the first model and the second feature output by the second model can both represent the frame style feature, the retrieval accuracy can be improved when performing frame style retrieval.

[0113] The embodiment of the present invention obtains the framework style features corresponding to the search object by inputting the search object into a feature extraction model corresponding to the type of the search object, and the feature extraction model includes the first model or the second model trained by the above-mentioned model training method. The framework style features are input into the first database for matching to obtain the search results corresponding to the search object; wherein the types of the search objects include pictures or framework components, the first database is a database related to the types of the search results, and the types of the search results include pictures or framework components. The embodiment of the present invention can realize image retrieval based on pictures, image retrieval of framework components, framework component retrieval of pictures, and framework component retrieval of framework components, and the embodiment of the present invention has a high retrieval accuracy.

[0114] refer to Figure 7 , Figure 7It is a schematic diagram of a model training process provided by an application embodiment of the present invention. The model training process includes: training image samples as training data of the yolo-v3 model, and the training image samples are page design drawings containing border information, such as the front-end design page drawings of application software. The border information is the label of the training image sample, and the elements in the training image sample are marked with a bounding box. The border information includes information such as the size of the bounding box and the type of the element in the bounding box. The training data of the simplified version of the yolo-v3 model is based on the tensor data obtained by converting the border information in the training image sample. That is, the training image sample is used as the training data of the yolo-v3 model, and the tensor data obtained by converting the border information in the training image sample is used as the training data of the simplified version of the yolo-v3 model.

[0115] During the training process, the image coding features (corresponding to the first feature) output by the yolo-v3 model and the semantic coding features (corresponding to the second feature) output by the simplified yolo-v3 model are obtained, the similarity between the image coding features and the semantic coding features is calculated, the loss value of the matching L2 loss function is calculated according to the similarity, and the weight parameters of the yolo-v3 model and / or the simplified yolo-v3 model are updated based on the loss value. Among them, the yolo-v3 model and the simplified yolo-v3 model share the output layer, obtain the prediction result of the yolo-v3 model for the image bounding boxes, and compare it with the target value. If the prediction result of the yolo-v3 model is too far from the target value, the model parameters of the yolo-v3 model are adjusted based on the image detection loss function, and the yolo-v3 model is retrained until the image detection loss function converges. Get the prediction results of the frame semantic bounding boxes of the simplified yolo-v3 model, and compare the prediction results of the simplified yolo-v3 model with the target value. If the prediction results of the simplified yolo-v3 model are too far from the target value, adjust the model parameters of the simplified yolo-v3 model based on the semantic detection loss function, and retrain the simplified yolo-v3 model until the semantic detection loss function converges.

[0116] refer to Figure 8 , Figure 8 It is a schematic diagram of a retrieval process provided by an application embodiment of the present invention. The retrieval process includes: firstly, constructing a retrieval database, using a trained first model to extract a first feature of a picture stored in a picture database to construct an image feature database, and using a second model to extract a second feature of a framework component in a framework component database to construct a framework component feature database.

[0117] The image feature database stores at least two images and a first feature corresponding to each of the at least two images. The first feature corresponding to each image in the image database is extracted in advance using a first model, and each image in the image database and the corresponding first feature are stored in the image feature database. The framework component feature database stores at least two framework components and a second feature corresponding to each of the at least two framework components. The second feature corresponding to each framework component in the framework component database is extracted in advance using a second model, and each framework component in the framework component database and the corresponding second feature are stored in the framework component feature database.

[0118] In the case of retrieving pictures based on pictures, first input the picture (Query picture) into the first model to extract the first feature, and input the first feature into the image feature database for matching to obtain the retrieval result. If the framework component is retrieved based on the picture, the input picture (Query picture) is input into the first model to extract the first feature, and the first feature is input into the framework component feature database for matching to obtain the retrieval result. If the picture is retrieved based on the framework component, the style information of the framework component is converted into tensor data, the tensor data is input into the second model to extract the second feature, and the second feature is input into the image feature database for matching to obtain the retrieval result. If the framework component is retrieved based on the framework component, the style information of the framework component is converted into tensor data, the tensor data is input into the second model to extract the second feature, and the second feature is input into the framework component feature database for matching to obtain the retrieval result.

[0119] The application of the present invention can realize image retrieval based on images, image retrieval based on framework components, framework component retrieval based on images, and framework component retrieval based on framework components, and the retrieval accuracy of the application embodiment of the present invention is high.

[0120] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0121] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0122] It should be noted that the technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

[0123] In addition, in the embodiments of the present invention, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0124] refer to Fig. 9 , Fig. 9 is a schematic diagram of a model training device provided by an embodiment of the present invention, such as Fig. 9 As shown, the device includes: a first input module, a second input module, a determination module and an update module.

[0125] A first input module is used to input a training image sample into a first model to obtain a first feature of the training image sample; the training image sample represents a page design image containing border information; the first feature represents a frame style feature of the border information in the training image sample;

[0126] A second input module is used to input the frame component sample into a second model to obtain a second feature of the frame component sample; the frame component sample is tensor data obtained by converting the border information in the training image sample; the second feature represents the frame style feature corresponding to the frame component sample;

[0127] A determination module, configured to determine a loss value corresponding to a set loss function based on a similarity between the first feature and the second feature;

[0128] An updating module is used to update the weight parameters of the first model and / or the second model according to the loss value.

[0129] The first model and the second model share an output layer.

[0130] The first model and the second model are constructed based on the YOLO model.

[0131] refer to Fig.10 , Fig.10 is a schematic diagram of a retrieval device provided by an embodiment of the present invention, such as Fig.10 As shown, the device includes: a third input module and a fourth input module.

[0132] A third input module is used to input the search object into a feature extraction model corresponding to the type of the search object to obtain a frame style feature corresponding to the search object; the feature extraction model includes a first model or a second model trained based on a model training method;

[0133] The fourth input module is used to input the frame style feature into the first database for matching, and obtain the search result corresponding to the search object; wherein,

[0134] The types of the search objects include pictures or framework components; the first database is a database related to the types of search results; the types of the search results include pictures or framework components.

[0135] In the case where the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; and the fourth input module is specifically used for:

[0136] Determine the similarity between the frame style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the frame style feature;

[0137] The search result is determined based on the similarity.

[0138] In the case where the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; and the fourth input module is specifically used for:

[0139] Determine the similarity between the frame style feature and each second feature stored in the first database; the first database stores at least two frame components and the second feature corresponding to each of the at least two frame components; the second feature represents the frame style feature;

[0140] The search result is determined based on the similarity.

[0141] In the case where the search object is a framework component, the feature extraction model corresponding to the type of the search object is the second model; and the fourth input module is specifically used for:

[0142] Determine the similarity between the frame style feature and each second feature stored in a first database; the first database stores at least two frame components and the second feature corresponding to each of the at least two frame components; the second feature represents the frame style feature;

[0143] The search result is determined based on the similarity.

[0144] In the case where the search object is a framework component, the feature extraction model corresponding to the type of the search object is the second model; and the fourth input module is specifically used for:

[0145] Determine the similarity between the frame style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the frame style feature;

[0146] The search result is determined based on the similarity.

[0147] In actual application, the first input module, the second input module, the determination module and the update module can be implemented by a processor in an electronic device, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a programmable gate array (FPGA).

[0148] It should be noted that: the model training device provided in the above embodiment only uses the division of the above modules as an example when performing model training. In practical applications, the above processing can be assigned to different modules as needed, that is, the internal structure of the device can be divided into different modules to complete all or part of the processing described above. In addition, the model training device provided in the above embodiment and the model training method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0149] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Fig.11 Schematic diagram of the hardware structure of the electronic device of the present application embodiment. Fig.11 As shown, the electronic equipment includes:

[0150] Communication interface, which can exchange information with other devices such as network equipment;

[0151] The processor is connected to the communication interface to realize information exchange with other devices, and is used to execute the method provided by one or more technical solutions of the electronic device side when running the computer program. The computer program is stored in the memory.

[0152] Of course, in actual applications, the various components in the electronic device are coupled together through the bus system. It is understandable that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.11 In the specification, various buses are labeled as bus systems.

[0153] The memory in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0154] It can be understood that the memory can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 130 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0155] The method disclosed in the above embodiment of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the program in the memory and completes the steps of the above method in combination with its hardware.

[0156] Optionally, when the processor executes the program, the corresponding processes implemented by the electronic device in each method of the embodiments of the present application are implemented, which will not be described in detail here for the sake of brevity.

[0157] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory storing a computer program, and the computer program can be executed by a processor of an electronic device to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0159] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0160] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0161] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0162] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0163] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0164] In addition, in the examples of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0165] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: The method comprises: Input the training image sample into the first model to obtain the first feature of the training image sample; the training image sample represents the page design image containing border information; the first feature represents the frame style feature of the border information in the training image sample; Inputting the frame component sample into the second model to obtain a second feature of the frame component sample; the frame component sample is tensor data obtained by converting the border information in the training image sample; the second feature represents the frame style feature corresponding to the frame component sample; Determining a loss value corresponding to a set loss function based on a similarity between the first feature and the second feature; Update weight parameters of the first model and / or the second model according to the loss value.

2. The method according to claim 1, characterized in that The first model and the second model share an output layer.

3. The method according to claim 1, characterized in that The first model and the second model are constructed based on the YOLO model.

4. A search method, characterized in that: The method comprises: Inputting a search object into a feature extraction model corresponding to the type of the search object to obtain a frame style feature corresponding to the search object; the feature extraction model comprises a first model or a second model trained by the model training method according to any one of claims 1 to 3; The frame style feature is input into the first database for matching to obtain the search result corresponding to the search object; wherein, The types of the search objects include pictures or framework components; the first database is a database related to the types of search results; the types of the search results include pictures or framework components.

5. The method according to claim 4, characterized in that In the case where the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; the step of inputting the frame style feature into the first database for matching to obtain a search result corresponding to the search object includes: Determine the similarity between the frame style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the frame style feature; The search result is determined based on the similarity.

6. The method according to claim 4, characterized in that In the case where the search object is a picture, the feature extraction model corresponding to the type of the search object is the first model; the step of inputting the frame style feature into the first database for matching to obtain a search result corresponding to the search object includes: Determine the similarity between the frame style feature and each second feature stored in the first database; the first database stores at least two frame components and the second feature corresponding to each of the at least two frame components; the second feature represents the frame style feature; The search result is determined based on the similarity.

7. The method according to claim 4, characterized in that In the case where the search object is a frame component, the feature extraction model corresponding to the type of the search object is the second model; the step of inputting the frame style feature into the first database for matching to obtain a search result corresponding to the search object includes: Determine the similarity between the frame style feature and each second feature stored in a first database; the first database stores at least two frame components and the second feature corresponding to each of the at least two frame components; the second feature represents the frame style feature; The search result is determined based on the similarity.

8. The method according to claim 4, characterized in that In the case where the search object is a frame component, the feature extraction model corresponding to the type of the search object is the second model; the step of inputting the frame style feature into the first database for matching to obtain a search result corresponding to the search object includes: Determine the similarity between the frame style feature and each first feature stored in the first database; the first database stores at least two pictures and the first feature corresponding to each of the at least two pictures; the first feature represents the frame style feature; The search result is determined based on the similarity.

9. A model training device, characterized in that: include: A first input module, used for inputting a training image sample into a first model to obtain a first feature of the training image sample; The training image sample represents a page design image containing border information; the first feature represents a frame style feature of the border information in the training image sample; A second input module is used to input the frame component sample into a second model to obtain a second feature of the frame component sample; the frame component sample is tensor data obtained by converting the border information in the training image sample; the second feature represents the frame style feature corresponding to the frame component sample; A determination module, configured to determine a loss value corresponding to a set loss function based on a similarity between the first feature and the second feature; An updating module is used to update the weight parameters of the first model and / or the second model according to the loss value.

10. A search device, characterized in that: include: A third input module is used to input the search object into a feature extraction model corresponding to the type of the search object to obtain a frame style feature corresponding to the search object; The feature extraction model includes a first model or a second model trained by the model training method according to any one of claims 1 to 3; The fourth input module is used to input the frame style feature into the first database for matching, and obtain the search result corresponding to the search object; wherein, The types of the search objects include pictures or framework components; the first database is a database related to the types of search results; the types of the search results include pictures or framework components.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model training method described in any one of claims 1 to 3 or the retrieval method described in any one of claims 4 to 8 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the model training method as described in any one of claims 1 to 3 or the retrieval method as described in any one of claims 4 to 8.

Images