Model iteration method and device, storage medium and electronic device

Abstract

The present disclosure relates to a model iteration method, device, storage medium and electronic equipment for automatically iterating a subsequent model of a cascade model and improving the iteration efficiency of the cascade model. The method comprises: receiving an iteration request for a target model, the iteration request carrying an identification of the target model and an identification of a sub-model; obtaining training data of the target model, calling the sub-model according to the identification of the sub-model, inputting the training data into the sub-model, and obtaining a prediction result feature output by the sub-model; calling the target model according to the identification of the target model, inputting the prediction result feature into the target model, and obtaining a target prediction result output by the target model; adjusting parameters of the target model according to the target prediction result, and obtaining an iteration model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a model iteration method, apparatus, storage medium, and electronic device. Background Technology

[0002] A cascaded model is a model in which the output of the previous layer serves as the input to the next layer, and the final layer outputs the final result. In a cascaded model, when the sub-model of the previous layer changes, the model of the next layer needs to be iterated.

[0003] The relevant techniques typically involve manually iterating the next layer of the model after changes to the previous layer. When multiple sub-models change frequently, the next layer model needs frequent iterations, which consumes a lot of manpower and time, resulting in low iteration efficiency. On the other hand, manually iterating the next layer model is time-consuming, causing iteration delays and affecting the real-time processing of online business. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, this disclosure provides a model iteration method, the method comprising:

[0006] Receive an iteration request for a target model. The iteration request is generated after a change to a sub-model cascaded with the target model. The input data of the target model is the output data of the sub-model. The iteration request carries the identifier of the target model and the identifier of the sub-model.

[0007] The training data of the target model is obtained, and the sub-model is called according to the identifier of the sub-model. The training data is input into the sub-model to obtain the prediction result features output by the sub-model.

[0008] The target model is invoked according to the identifier of the target model, and the prediction result features are input into the target model to obtain the target prediction result output by the target model;

[0009] The parameters of the target model are adjusted based on the target prediction results to obtain an iterative model.

[0010] Secondly, this disclosure provides a model iteration apparatus, the apparatus comprising:

[0011] A receiving module is used to receive an iteration request for a target model. The iteration request is generated after a change has been made to a sub-model cascaded with the target model. The input data of the target model is the output data of the sub-model, and the iteration request carries the identifier of the target model and the identifier of the sub-model.

[0012] The first training module is used to acquire training data and call the sub-model according to the identifier of the sub-model, input the training data into the sub-model, and obtain the prediction result features output by the sub-model, wherein the training data is labeled with labels used to characterize the sample results;

[0013] The second training module calls the target model according to the identifier of the target model, and inputs the prediction result features into the target model to obtain the target prediction result output by the target model;

[0014] The third training module is used to adjust the parameters of the target model based on the target prediction results to obtain an iterative model.

[0015] Thirdly, this disclosure provides a non-transitory computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in the first aspect.

[0016] Fourthly, this disclosure provides an electronic device, comprising:

[0017] A storage device on which computer programs are stored;

[0018] A processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect.

[0019] The above technical solution involves an iteration request carrying the identifiers of both the target model and its sub-models. Furthermore, the iteration request is generated after changes to the sub-models cascaded with the target model. Therefore, after a sub-model change, the model can be iterated based on its corresponding identifier, enabling automatic iteration of the next layer in the cascaded model. Compared to manual iteration in related technologies, this reduces the manpower and time required for cascaded model iteration, improving its efficiency. On the other hand, it allows for faster invocation of the iterated model for business processing, minimizing the impact of delayed model iteration on business operations.

[0020] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0022] Figure 1 This is a schematic diagram of a cascaded model according to an exemplary embodiment of the present disclosure;

[0023] Figure 2 This is a flowchart illustrating a model iteration method according to an exemplary embodiment of the present disclosure;

[0024] Figure 3 This is a schematic diagram illustrating a model iteration method according to an exemplary embodiment of the present disclosure;

[0025] Figure 4 yes Figure 3 The diagram illustrates the task identifier query process in the model iteration method shown.

[0026] Figure 5 yes Figure 3 The diagram shows the task scheduling process in the model iteration method.

[0027] Figure 6 This is a block diagram illustrating a model iteration apparatus according to an exemplary embodiment of the present disclosure;

[0028] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0036] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0037] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0039] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0040] In various business operations, it is often necessary to classify content such as videos, texts, images, and audios into different categories to meet business needs, and machine learning models are usually built to automatically classify the content. However, in complex business scenarios, it is difficult to directly classify the content and it is difficult to achieve business targets. Therefore, the categories that need to be classified are decomposed according to their features or attributes, and then sub-models are built separately for the features or attributes obtained from the decomposition to classify them. Finally, the outputs of each sub-model are input into a fusion model to obtain the final classification result. For example, to classify restaurant reviews to distinguish whether they are positive or negative, the positive and negative categories can be divided into the following five categories: (1) Is parking convenient? (2) Is the environment beautiful? (3) Is it hygienic? (4) Is the restaurant noisy? (5) Is the food delicious? Then, sub-models are built to determine the classification probability of the above five categories. Finally, the classification probabilities output by these five sub-models are input into a fusion model to obtain the classification result corresponding to the review.

[0041] The model constructed in the above scenario is called a cascaded model, where the output of the previous layer serves as the input to the next layer, and the final layer outputs the final result. For example... Figure 1 As shown, the outputs of the previous layer sub-models 1, 2, 3, ..., N (N is a positive integer) serve as the input to the next layer fusion model. Finally, business decisions are made based on the output of the fusion model, resulting in strategy 1, strategy 2, ..., strategy K (K is a positive integer). If a previous layer sub-model changes, the next layer model needs to be iterated; otherwise, the final model's output category will be unbelievable. However, current techniques typically involve manually iterating the next layer model after a change in the previous layer sub-model. When multiple sub-models frequently change, the next layer model requires frequent iterations, consuming significant manpower and time, resulting in low iteration efficiency. Furthermore, manual iteration of the next layer model is time-consuming, causing iteration delays and impacting real-time processing of online business.

[0042] Another related technology involves iterating on the next-layer model. This method primarily involves normalizing the classification probability values ​​output by the modified sub-model according to a certain business metric. If the distribution of the normalized data is not significantly different from the distribution of the original training data of the next-layer fusion model, then the next-layer fusion model is deemed to require no modification. However, this method is only applicable to specific business scenarios. The normalized data can only approximate the original training set in terms of distribution, and there is still a certain difference between it and the true data classification. Therefore, the final output classification also differs from the true classification, failing to effectively achieve iteration of the fusion model.

[0043] In view of this, the present disclosure provides a model iteration method, apparatus, storage medium, and electronic device to automatically iterate the next layer of a cascaded model, thereby improving the iteration efficiency of the cascaded model.

[0044] Figure 2 This is a flowchart illustrating a model iteration method according to an embodiment of the present disclosure. (Refer to...) Figure 2 This model iterative method can be applied to servers, including:

[0045] Step 201: Receive an iteration request for the target model. The iteration request is generated after changes to the sub-model cascaded with the target model. The input data of the target model is the output data of the sub-model, and the iteration request carries the identifiers of both the target model and the sub-model.

[0046] Step 202: Obtain training data and call the sub-model according to its identifier. Input the training data into the sub-model to obtain the predicted feature output by the sub-model. The training data is labeled with tags used to characterize the sample results.

[0047] Step 203: Invoke the target model according to the target model's identifier, and input the prediction result features into the target model to obtain the target prediction result output by the target model.

[0048] Step 204: Adjust the parameters of the target model based on the target prediction results and labels corresponding to the training data to obtain the iterative model.

[0049] In this method, the iteration request carries the identifiers of the target model and the sub-model. Furthermore, the iteration request is generated after changes to the sub-model cascaded with the target model. Therefore, after a sub-model changes, the model can be invoked for iteration based on the corresponding identifier, achieving automatic iteration of the next layer model in the cascaded model. Compared to manual iteration methods in related technologies, this reduces the manpower and time required for cascaded model iteration, improving its iteration efficiency. On the other hand, it allows for faster invocation of the iterated model for business processing, reducing the impact of delayed model iteration on business operations.

[0050] To enable those skilled in the art to better understand the model iteration method provided in this disclosure, the above steps are illustrated in detail below.

[0051] For example, the target model in step 201 can be the next layer model in a cascaded model, such as Figure 1 The fusion model shown. The sub-model cascaded with the target model can be a previous layer model in the cascaded model, such as... Figure 1 The sub-models shown are sub-model 1, sub-model 2, ..., sub-model N (N is a positive integer). That is to say, in this embodiment, the input data of the target model is the output data of the sub-model. Furthermore, it should be understood that the model iteration model provided in this disclosure can be applied to, for example... Figure 1 The iteration of the latter layer in the two-layer cascaded model shown can also be applied to the iteration of other layers in a three-layer or more cascaded model, excluding the first layer. In practical applications, such as in video classification scenarios, the sub-models may include at least one of the following models: an image classification model for video frame classification, a text classification model for text classification in video, and a speech recognition model for speech recognition in video. Correspondingly, the target model may be a fusion classification model that classifies based on the results output by the at least one sub-model. This disclosure does not limit the specific form and structure of the target model and sub-models, and they can be set according to the actual situation.

[0052] For example, an iteration request may be generated after a change occurs to a sub-model cascaded with the target model. This change can be any of the following: sub-model iteration, sub-model decommissioning, or a new sub-model being added. The iteration request may be generated and sent to the server after relevant personnel manually trigger model iteration on the front-end page following a sub-model change; alternatively, the iteration request may be automatically generated and sent to the server when the business platform detects that the output of the target model does not meet preset business indicators; or the iteration request may be automatically generated and sent to the server by the front-end page in response to a pre-set scheduled iteration task. This embodiment of the present disclosure does not limit the scope of the requests.

[0053] For example, the iteration request may carry the identifier of the target model and the identifier of the sub-models. The identifiers of the target model and the sub-models may be input by relevant personnel before triggering the iteration task or when setting a timed iteration task, or they may be automatically obtained when the output of the target model is detected to not meet preset business indicators; this disclosure does not limit this. Furthermore, since the input of the target model includes the output of each sub-model, the identifier of the sub-model includes the identifier of each sub-model cascaded with the target model.

[0054] It should be understood that the iterative process of the subsequent layer in a cascaded model includes: data backtracking of the previous layer and training of the subsequent layer. Data backtracking of the previous layer refers to obtaining the output data of each sub-model to obtain the input data of the subsequent layer. Therefore, in this embodiment, upon receiving an iteration request, training data is first obtained, and then the sub-model is invoked according to its identifier, with the training data input into the sub-model to obtain the predicted feature output by the sub-model. Thus, data backtracking can be automatically completed based on the sub-model identifier carried in the iteration request, thereby improving iteration efficiency.

[0055] For example, the iteration request may include preset time information. Accordingly, a corresponding time range can be determined first based on this preset time information, and then business data labeled with the corresponding tags within that time range can be obtained from the business system. For instance, in a video classification scenario, video data labeled with classification tags within the time range corresponding to the preset time information can be obtained from the business system. In another possible approach, the labeled business data can be stored in a first offline table, so that training data can be obtained from this first offline table after receiving the iteration request.

[0056] After obtaining the training data, the sub-model can be invoked based on its identifier, and the training data can be input into the sub-model to obtain the predicted feature output by the sub-model. This predicted feature is the output data of the sub-model based on the training data, which is also the input data of the target model. Therefore, the target model can then be invoked based on its identifier, and the predicted feature can be input into the target model to obtain the target prediction result output by the target model. Alternatively, after obtaining the predicted feature output by the sub-model, this predicted feature can be stored in a second offline table. After invoking the target model, the predicted feature can be retrieved from the second offline table and input into the target model.

[0057] For example, since the training data is pre-labeled, a loss function value can be calculated based on the target prediction result corresponding to the training data and the label. The parameters of the target model can then be adjusted based on this loss function value, making the target prediction result output by the target model increasingly closer to the sample result corresponding to the label. This yields an iterative model. It should be understood that the specific parameter adjustment process for the target model is similar to related techniques and will not be elaborated upon here.

[0058] In one possible approach, step 203 could be: verifying whether the prediction result features meet preset conditions; if the prediction result features do not meet preset conditions, then determining the retry sub-model and retry training data corresponding to the prediction result features; re-calling the retry sub-model according to the identifier of the retry sub-model; inputting the retry training data into the retry sub-model to obtain the retry result features; and then inputting the retry result features and the prediction result features output by other sub-models besides the retry sub-model into the target model to obtain the target prediction result output by the target model.

[0059] It should be understood that in practical applications, feature missing issues may occur due to sub-model call failures or other errors, resulting in incomplete data input to the target model. This affects the normal iteration of the target model and the accuracy of the output results after iteration. Therefore, embodiments of this disclosure provide a feature completion method to make the output results of the target model after iteration more reliable.

[0060] For example, the preset conditions could be that the predicted feature is not missing or the predicted feature is not expired. Whether the predicted feature is missing can be determined by whether the predicted feature output from each sub-model cascaded with the target model is obtained. Whether the predicted feature is expired can be determined by checking whether the version number of each sub-model is the latest version number.

[0061] For example, if the sub-model's first prediction feature output for a certain training data is missing, the same training data can be reacquired and then re-input into the target sub-model to obtain the second prediction feature output from the target sub-model for that training data. This second prediction feature is the retry result feature. Alternatively, if the sub-model's version number is not the latest version, the same approach can be taken: reacquire the training data and input it into the sub-model to obtain the same prediction feature output from the sub-model for that training data. This prediction feature is also the retry result feature.

[0062] Finally, the retry result features and the prediction result features output by other sub-models (excluding the retry sub-model) are input into the target model to obtain the target prediction result output by the target model. This improves the accuracy and timeliness of the prediction result features, allowing the target model in the next layer to obtain more confident results during training. On the other hand, each prediction result feature corresponds to a piece of training data. Related technologies typically discard the prediction result feature and its corresponding training data directly when the feature is missing or expired. However, in this embodiment, a retry mechanism is implemented when a prediction result feature is missing or expired, allowing the training data corresponding to that feature to be reused in the iteration process, thereby reducing the training data discard rate.

[0063] In one possible approach, the iteration request carries the target task identifier for this iteration task. Step 202 could be: first, determine whether a target task identifier exists in the task queue, which stores task identifiers for the target model's iteration tasks in chronological order. If the target task identifier does not exist in the task queue, then associate the target task identifier with the submission time of the iteration request and write it into the task queue. Finally, a polling thread traverses the task queue. When the target task identifier is encountered, the training data for the target model is obtained. This polling thread traverses the task identifiers in the task queue in chronological order.

[0064] For example, the target task identifier can be determined by the time the iteration request is initiated, and this embodiment of the disclosure does not limit this. In this embodiment of the disclosure, the server can maintain a task queue, which can store task identifiers in the order of the submission time of the iteration request. On the other hand, the server provides a polling thread for traversing the task queue in chronological order. This polling thread executes continuously and can promptly monitor the iteration tasks to be executed. When the server receives an iteration request, it first queries the task queue based on the target task identifier included in the iteration request. When the target task identifier is found in the task queue, it means that the same iteration task has been executed before. When the polling thread traverses to the target task identifier, it can obtain the historical data of the previous execution and re-iterate the target model, that is, retry for the target model. If the target task identifier is not found in the task queue, it means that the same iterative task has not been executed before. The target task identifier can be associated with the submission time of the iteration request and written into the task queue. When the polling thread traverses to the target task identifier, it can start executing the iterative task for the target model, that is, start executing steps 202 to 204. Otherwise, continue traversing.

[0065] In one possible approach, this model iteration method can be applied to the target server. Before step 201, when the target task identifier is encountered, it can be determined whether the server information associated with the target task identifier is empty. This server information is used to represent the information of the server executing this iteration task. If the server information associated with the target task identifier is empty, the server information corresponding to the target task identifier can be set to the information of the target server.

[0066] It should be understood that in practical applications, multiple servers can be deployed for model iteration. These servers can execute iteration tasks for different target models in parallel, thereby improving the efficiency of model iteration in business scenarios. Furthermore, these multiple servers can jointly maintain a task queue. Therefore, to reduce execution errors in iteration tasks, before each server executes an iteration task, when a server iterates through the task queue using its own polling thread and encounters a target task identifier, if the server information associated with that target task identifier is empty, it means no server is executing the iteration task corresponding to that target task identifier. In this case, the server information associated with the target task identifier can be set to its own server information, such as setting the machine IP of the target task identifier to its own server IP, before starting to acquire training data for model iteration. When other servers iterate through a target task identifier, if the server information associated with that target task identifier is not empty, it means a server has already executed the iteration task corresponding to that target task identifier, and the other servers will continue to iterate through other task identifiers in the task queue. This avoids multiple servers executing the same iteration task, ensuring the normal progress of model iteration.

[0067] It should also be understood that if, after traversing to the target task identifier, the server determines that the server information associated with that target task identifier is its own server information, it means that the server previously executed the iterative task corresponding to that target task identifier, but the execution result was a failure, and a retry is required. This retry process can either re-execute the entire model iteration process or continue executing the model iteration process based on the steps that were successful in the previous execution; this disclosure does not limit this approach.

[0068] In one possible approach, if a target task identifier exists in the task queue, historical step information associated with the target task identifier can be retrieved from the cache. This historical step information represents the historical steps that have been successfully executed during the historical iterations of the target model. Then, based on the historical steps, the target iteration step to be executed in the current iteration is determined. This target iteration step is the next step after the historical steps described in the preset iteration flow. Finally, intermediate iteration data obtained by executing the historical steps is acquired, and the target iteration step is executed based on the intermediate iteration data to obtain the iterative model.

[0069] For example, caching can be implemented using a Redis database, and this disclosure does not limit this embodiment. In this disclosure embodiment, historical iteration data can be stored in the cache, such as iteration task identifiers, iteration parameters, iteration status, step information, and server information during the historical iteration process. The step information can include the execution status and result of each historical step, and the server information can be the IP address of the server executing the iteration task. Therefore, when a target task identifier is found in the task queue, it indicates that the same iteration task has been executed previously. The corresponding historical step information can then be retrieved from the cache to continue executing the iteration task from that historical step information, instead of re-executing all iteration processes, thereby improving iteration efficiency.

[0070] For example, the target iteration step to be executed in the current iteration can be determined first based on historical step information. This target iteration step is the next step in the preset iteration process. The preset iteration process can sequentially include the sub-model data backtracking step and the target model training step. Alternatively, the preset iteration process can sequentially include the sub-model data backtracking step, the target model input feature completion step, and the target model training step. Or, the preset iteration process can sequentially include the sub-model data backtracking step, the target model input feature completion step, the target model training and evaluation step, the evaluation report and model generation step, and the one-click model deployment step. Here, sub-model data backtracking is the step of obtaining the prediction result features; target model input feature completion is the retry step executed when the verification prediction result features do not meet preset conditions; target model training is the step of obtaining the target prediction result and adjusting the target model parameters to obtain the iterative model; target model evaluation is the step of evaluating the iterative model using labeled validation set data; generating an evaluation report is the step of outputting a report based on the evaluation results for the target model; and one-click model deployment is the step of deploying the iterative model through user operations on the front-end page. In practical applications, preset iteration processes can be set according to business needs, and this disclosure does not limit this.

[0071] After determining the target iteration step, intermediate iteration data obtained by executing historical steps can be acquired, and the target iteration step can be executed based on the intermediate iteration data to obtain the iteration model.

[0072] For example, during each iteration, the data obtained through each iteration step can be saved locally. Correspondingly, intermediate iteration data obtained by executing historical steps can be loaded locally. For instance, in a certain iteration task, during the sub-model data backtracking step, 40,000 data points were backtracked and stored locally on the server; that is, the predicted feature data corresponding to 40,000 training data points is stored locally. If a retry is needed for this iteration task, these 40,000 predicted feature data points can be loaded locally first, then input into the target model, and the target model training steps can continue. Therefore, continuing iteration from a previously successfully executed step can improve iteration efficiency.

[0073] It should be understood that after each step is executed, the historical step information in the cache can be updated based on the execution result. Additionally, after successfully executing all steps corresponding to an iterative task, the target task identifier associated with that iterative task can be deleted from the task queue; otherwise, the target task identifier can be retained in the task queue for a subsequent retrieval.

[0074] After obtaining the iterative model, the obtained iterative model can be put online.

[0075] In one possible approach, in response to a deployment request for the iterative model, the target model running on the business platform can be replaced with the iterative model, and the business service associated with the target model in the business platform can be restarted. This allows the restarted business service to call the iterative model for business processing. The deployment request is generated in response to a user's triggering action on the deployment control in the front-end page.

[0076] For example, the online control is a control displayed on the front-end page that allows users to trigger the online operation. This embodiment of the disclosure does not limit the form or display method of the online control. When a user triggers the online control on the front-end page, the front-end page can generate an online request and send it to the back-end server. After receiving the online request, the server can replace the target model running on the business platform with the iterative model and restart the business service associated with the target model in the business platform, so that the restarted business service can call the iterative model for business processing. Alternatively, after obtaining the iterative model, it can be stored in a model storage system. When the server receives the online request, it can download the iterative model and its corresponding configuration information from the model storage system. If the corresponding version of the iterative model and configuration are successfully downloaded, the original model and configuration of the business platform are replaced, and the business service associated with the target model in the business platform is automatically restarted, and the corresponding configuration of the target model in the business platform is automatically upgraded. If the download fails, the original business service remains unchanged. When the new business service restarts, a notification of successful restart can also be sent to the user who initiated the online operation. This embodiment of the disclosure does not limit this aspect.

[0077] This standardizes the entire model deployment process, enabling non-algorithm-related business personnel to iterate and deploy models, thus improving efficiency. Furthermore, deploying models by restarting the business services associated with the target model is simple, efficient, and has minimal impact on the stability of those services.

[0078] In one possible approach, the online request may carry the target model threshold entered by the user on the front-end page. This target model threshold is used to output the results of the iterative model. After replacing the target model running on the business platform with the iterative model, the model threshold of the iterative model can also be determined and replaced with the target model threshold, so that the business service can perform business processing based on the target model threshold after calling the iterative model.

[0079] For example, a model threshold is used to determine the corresponding result of the model output. In a classification scenario, when the probability value output by the classification model is greater than the model threshold, it can be determined that the model belongs to the first category; when the probability value output by the classification model is less than or equal to the model threshold, it can be determined that the model belongs to the second category, and so on. Different model thresholds can be set to obtain corresponding model output results. Related technologies typically automatically determine a model threshold, which is optimal for model output results, but may not be suitable for actual business scenarios, thus failing to obtain good business metrics. Therefore, this disclosure provides a method for modifying the model threshold. Before initiating a deployment request, business personnel can input a custom target model threshold on the front-end page. After initiating the deployment request, the model can be deployed and iterated according to the custom target model threshold to meet the business needs of different business scenarios.

[0080] The model iteration method provided in this disclosure is illustrated below through another embodiment.

[0081] by Figure 1 Taking the cascaded model shown as an example, the fusion model of the last layer is iterated, referring to... Figure 3First, it determines whether a sub-model change has occurred. If a sub-model change has occurred, model iteration begins. The model iteration process can include the following five steps: sub-model data backtracking, input feature completion for the fusion model, automatic training and evaluation of the fusion model, generation of evaluation reports and the model, and one-click model deployment. Additionally, three offline tables are provided. Offline table one stores the training and validation set data of the fusion model. Both training and validation set data are labeled with results. The training set data is used to train the fusion model to obtain the iterative model, and the validation set data is used to verify the performance of the iterative model. Offline table two stores the output results of each version of a single sub-model. Offline table three stores the output results of each version of all sub-models; that is, the data stored in offline table three is essentially a summary of all data in offline table two.

[0082] When a sub-model changes, an iteration request is generated and sent to the server. Upon receiving the request, the server retrieves the target task identifier for the current iteration, the preset time information for data backtracking, the identifier of the fusion model, and the identifier of the sub-model. First, training set data is retrieved from offline table one. Then, the training set data is input into the sub-model for data backtracking, yielding the sub-model's output. This output is stored in offline table two and then aggregated in offline table three. After sub-model data backtracking, information such as data ID, data category, and sub-model identifier can be read from offline table three. The server automatically verifies which sub-model outputs are missing or outdated, thus determining the retry training dataset and retry sub-models. Next, the retry sub-model is invoked based on its identifier, and data from the retry training dataset is input into it to obtain retry result features. The data in offline table three is then updated.

[0083] Next, we can proceed to the training and evaluation of the fusion model. During the training phase, prediction result features and retry result features can be read from offline Table 3. The fusion model is then invoked based on its identifier, and the read result features are input into the fusion model to obtain the target prediction result. The parameters of the fusion model are then adjusted based on this target prediction result to obtain the iterative model. First, the model parameters and related configuration information of the fusion model can be set. For example, model parameters may include the identifier of the fusion model, and related configuration information may include preset time information for data backtracking, target model thresholds, etc. This information can be read in real-time through a configuration file or left unchanged after the initial setting, depending on the business requirements. Then, machines for model training and evaluation can be deployed, such as CPU (Central Processing Unit) servers or GPU (Graphics Processing Unit) servers, depending on the different machine learning models. Alternatively, it can connect to an existing training platform and perform model training and evaluation through pre-deployed servers on that platform. Next, real-time monitoring of the scheduling task ID can be set up. When an iterative task is initiated, the task can be executed based on the monitored information.

[0084] like Figure 4 As shown, when an iteration request is received, the server's Trigger interface retrieves the Task ID (Taskid) and other parameter information carried in the iteration request and performs parameter validation. If the parameter validation passes, it queries whether there is a historical record based on the Task ID (Taskid). If the Task ID (Taskid) is found in the task queue, it returns to task execution; otherwise, it creates a new record, stores it in the queue, and returns to task execution. In this embodiment, the Redis cache can store the task queue and the historical iteration history. The relevant content of the task queue and historical history can be referred to above and will not be repeated here. In addition, the server also provides a Query interface for querying historical history on the front-end page. That is, when a user initiates a query request for historical iteration tasks on the front-end page, the Query interface is used to query Redis and return the results to the front-end page for display.

[0085] During the execution of the task, such as Figure 5As shown, a polling thread iterates through the task queue to find executable tasks. The polling thread can iterate through the task queue by time, querying the corresponding server IP. If the IP is empty, it indicates a new task; the server IP is set and execution begins. Otherwise, it further checks if the IP is the local machine's IP; if it is, a retry is performed. Next, the task date of the found executable tasks can be verified. If the task date is yesterday or a previous historical time, the status of the executable task is updated to timeout failure, and the corresponding task identifier is removed from the task queue, ending execution. If the task date verification passes, the task logic is initiated to determine if a historical record can be found. If not, the iterative task begins from the data backtracking step. If a historical record is found, the next step to be executed is located based on the step information in the historical record, skipping previously successfully executed steps. Then, each step is executed, and the record information in Redis is updated. Each step can retain the execution result and corresponding data locally. At the beginning of each step, it can attempt to load intermediate iteration data from the historical step locally. If loading is successful, iteration is performed based on this intermediate iteration data; if loading fails, data is retrieved from an offline table for iteration. Finally, the execution results are evaluated. If all are successful, the corresponding task identifier is removed from the task queue; otherwise, the corresponding task identifier is left in the task queue for retry, until the execution is successful or the number of retries reaches the preset number.

[0086] During the evaluation phase, validation set data can be read from offline table 1 to evaluate the iterative model. Once evaluation is complete, the task ID and evaluation report are automatically sent to the task initiator. The initiator can verify whether the evaluation metrics meet the requirements, thus determining whether the model for this iteration needs to be deployed. If the iterative model needs to be deployed, the initiator can perform a one-click deployment operation on the front-end page, or can initiate a one-click deployment operation after entering the target model threshold, making it convenient for the initiator to deploy the model according to the corresponding threshold.

[0087] By employing the methods described above, significant manpower costs can be saved in iterating fusion models, and the iteration time can be reduced from days to hours. In the face of rapidly changing business data and categories, automatic model iteration can quickly meet business needs, especially in the event of sudden information changes. Furthermore, one-click model deployment standardizes the entire deployment process, enabling non-algorithm-related business personnel to iterate and deploy models, thus improving iteration and deployment efficiency.

[0088] Based on the same concept, this disclosure also provides a model iteration apparatus, which can be part or all of an electronic device through software, hardware, or a combination of both. (See also...) Figure 6The model iteration device 600 includes:

[0089] The receiving module 601 is used to receive an iteration request for a target model. The iteration request is generated after a change has been made to a sub-model cascaded with the target model. The input data of the target model is the output data of the sub-model, and the iteration request carries the identifier of the target model and the identifier of the sub-model.

[0090] The first training module 602 is used to acquire training data and call the sub-model according to the identifier of the sub-model, input the training data into the sub-model, and obtain the prediction result features output by the sub-model, wherein the training data is labeled with labels used to characterize the sample results;

[0091] The second training module 603 calls the target model according to the identifier of the target model, and inputs the prediction result features into the target model to obtain the target prediction result output by the target model;

[0092] The third training module 604 is used to adjust the parameters of the target model based on the target prediction results to obtain an iterative model.

[0093] Optionally, the second training module 603 is used for:

[0094] Verify whether the predicted result features meet preset conditions;

[0095] When the predicted result feature does not meet the preset condition, determine the retry sub-model and retry training data corresponding to the predicted result feature, call the retry sub-model again according to the identifier of the retry sub-model, input the retry training data into the retry model, and obtain the retry result feature;

[0096] The retry result features and the prediction result features output by other sub-models besides the retry sub-model are input into the target model to obtain the target prediction result output by the target model.

[0097] Optionally, the iteration request carries the target task identifier for this iteration task, and the first training module 602 is used to:

[0098] Determine whether the target task identifier exists in the task queue, wherein the task queue is used to store task identifiers of iterative tasks to be executed for the target model in chronological order;

[0099] When the target task identifier does not exist in the task queue, the target task identifier is associated with the submission time of the iteration request and written into the task queue;

[0100] The polling thread iterates through the task queue. When the target task identifier is encountered, the training data of the target model is obtained. The polling thread is used to traverse the task identifiers in the task queue in chronological order.

[0101] Optionally, the apparatus is applied to a target server, and the apparatus 600 further includes:

[0102] The first determining module is used to determine whether the server information associated with the target task identifier is empty when the target task identifier is encountered during traversal before obtaining the training data of the target model. The server information is used to characterize the information of the server executing the current iteration task.

[0103] The setting module is used to set the server information corresponding to the target task identifier to the information of the target server when the server information associated with the target task identifier is empty.

[0104] Optionally, the device 600 further includes:

[0105] The acquisition module is used to acquire historical step information associated with the target task identifier from the cache when the target task identifier exists in the task queue. The historical step information is used to characterize the historical steps that have been successfully executed in the historical iteration process of the target model.

[0106] The second determining module is used to determine the target iteration step to be executed in the current iteration process based on the historical steps, wherein the target iteration step is the next step of the historical steps in the preset iteration process;

[0107] An execution module is used to obtain intermediate iteration data obtained by executing the historical steps, and to execute the target iteration steps based on the intermediate iteration data to obtain an iteration model.

[0108] Optionally, the device 600 further includes:

[0109] The restart module is used to respond to the online request for the iterative model, replace the target model running on the business platform with the iterative model, and restart the business service associated with the target model in the business platform, so that the restarted business service calls the iterative model to perform business processing. The online request is generated in response to the user's trigger operation on the online control in the front-end page.

[0110] Optionally, the online request carries a target model threshold input by the user on the front-end page, and the target model threshold is used in the output result of the iterative model. The device 600 further includes:

[0111] The replacement module is used to determine the model threshold of the iterative model after replacing the target model running on the business platform with the iterative model, and replace the model threshold of the iterative model with the target model threshold, so that the business service performs business processing based on the target model threshold after calling the iterative model.

[0112] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0113] Based on the same concept, this disclosure also provides a non-transitory computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of any of the above-described model iteration methods.

[0114] Based on the same concept, this disclosure also provides an electronic device, including:

[0115] A storage device on which computer programs are stored;

[0116] A processing device for executing the computer program in the storage device to implement the steps of any of the above-described model iteration methods.

[0117] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0118] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0119] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0120] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0121] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0122] In some implementations, communication can be conducted using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can be interconnected with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0123] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0124] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive an iteration request for a target model, wherein the iteration request is generated after a change to a sub-model cascaded with the target model, the input data of the target model is the output data of the sub-model, and the iteration request carries an identifier of the target model and an identifier of the sub-model; acquire training data of the target model, and call the sub-model according to the identifier of the sub-model, inputting the training data into the sub-model to obtain the prediction result features output by the sub-model; call the target model according to the identifier of the target model, and input the prediction result features into the target model to obtain the target prediction result output by the target model; and adjust the parameters of the target model according to the target prediction result to obtain an iterative model.

[0125] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0128] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0129] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0131] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0132] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. An iterative method for a classification model, characterized in that, The method includes: The system receives an iteration request for a target model, which is generated after a change to a sub-model cascaded with the target model. The input data of the target model is the output data of the sub-model, and the iteration request carries the identifier of the target model and the identifier of the sub-model. The sub-model is a model for classifying content, which includes at least one of video, text, image, and audio. The target model is a fusion classification model that classifies content based on the output of the sub-model. The change to the sub-model includes at least one of sub-model iteration, sub-model decommissioning, and adding a new sub-model. The training data of the target model is obtained, and the sub-model is called according to the identifier of the sub-model. The training data is input into the sub-model to obtain the prediction result features output by the sub-model. The target model is invoked according to the identifier of the target model, and the prediction result features are input into the target model to obtain the target prediction result output by the target model; The parameters of the target model are adjusted based on the target prediction results to obtain an iterative model.

2. The method according to claim 1, characterized in that, The step of inputting the predicted result features into the target model to obtain the target prediction result output by the target model includes: Verify whether the predicted result features meet preset conditions; If the predicted result feature does not meet the preset condition, the sub-model corresponding to the predicted result feature is determined as the retry sub-model, and the training data corresponding to the predicted result feature is determined as the retry training data. The retry sub-model is called again according to the identifier of the retry sub-model, and the retry training data is input into the retry sub-model to obtain the retry result feature. The retry result features and the prediction result features output by other sub-models besides the retry sub-model are input into the target model to obtain the target prediction result output by the target model.

3. The method according to claim 1, characterized in that, The iteration request carries the target task identifier for this iteration task, and obtaining the training data of the target model includes: Determine whether the target task identifier exists in the task queue, wherein the task queue is used to store task identifiers of iterative tasks to be executed for the target model in chronological order; If the target task identifier does not exist in the task queue, then the target task identifier is associated with the submission time of the iteration request and written into the task queue; The polling thread iterates through the task queue. When the target task identifier is encountered, the training data of the target model is obtained. The polling thread is used to traverse the task identifiers in the task queue in chronological order.

4. The method according to claim 3, characterized in that, The method is applied to the target server, and before acquiring the training data of the target model, the method further includes: When the target task identifier is encountered, it is determined whether the server information associated with the target task identifier is empty. The server information is used to represent the information of the server executing this iteration task. If the server information associated with the target task identifier is empty, then the server information corresponding to the target task identifier is set to the information of the target server.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If the target task identifier exists in the task queue, the historical step information associated with the target task identifier is retrieved from the cache. The historical step information is used to characterize the historical steps that have been successfully executed during the historical iteration of the target model. Based on the historical steps, the target iteration step to be executed in this iteration is determined, and the target iteration step is the next step of the historical steps in the preset iteration process; Obtain intermediate iteration data obtained by executing the historical steps, and execute the target iteration steps based on the intermediate iteration data to obtain the iteration model.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: In response to the request to deploy the iterative model, the target model running on the business platform is replaced with the iterative model, and the business service associated with the target model in the business platform is restarted, so that the restarted business service calls the iterative model to perform business processing; The online request is generated in response to the user's triggering operation on the online control in the front-end page.

7. The method according to claim 6, characterized in that, The online request carries a target model threshold input by the user on the front-end page. The target model threshold is used in the output result of the iterative model. After replacing the target model running on the business platform with the iterative model, the method further includes: A model threshold is determined for the iterative model, and the model threshold of the iterative model is replaced with the target model threshold, so that the business service performs business processing based on the target model threshold after calling the iterative model.

8. A classification model iteration device, characterized in that, The device includes: A receiving module is configured to receive an iteration request for a target model. The iteration request is generated after a change has been made to a sub-model cascaded with the target model. The input data of the target model is the output data of the sub-model. The iteration request carries the identifier of the target model and the identifier of the sub-model. The sub-model is a model for classifying content, and the content includes at least one of video, text, image, and audio. The target model is a fusion classification model that classifies content based on the output of the sub-model. The change of the sub-model includes at least one of sub-model iteration, sub-model decommissioning, and adding a new sub-model. The first training module is used to acquire training data and call the sub-model according to the identifier of the sub-model, input the training data into the sub-model, and obtain the prediction result features output by the sub-model, wherein the training data is labeled with labels used to characterize the sample results; The second training module calls the target model according to the identifier of the target model, and inputs the prediction result features into the target model to obtain the target prediction result output by the target model; The third training module is used to adjust the parameters of the target model based on the target prediction results to obtain an iterative model.

9. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processing device, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-7.

Images