A model search method, device, equipment, storage medium, and program product

Abstract

The present application provides a model search method, device, equipment, storage medium, and program product; the embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, vehicle-mounted, and medical treatment, and involve artificial intelligence technology; the method includes: based on the search data set, searching for a first weight of network granularity and a plurality of second weights of operator granularity for a plurality of network units in a target super network from a search space; according to the first weight, extracting at least two target network units from a plurality of network units in a target super network, and connecting the at least two target network units to obtain a target network structure; based on the plurality of second weights and a plurality of operation operators, generating a target operator corresponding to each target network unit of the target network structure; using the target operator and each target network unit, constructing a target network model, and completing the model search. Through the present application, the performance of model search can be improved.

Description

Technical Field

[0001] The present application relates to computer vision technology in the field of artificial intelligence, and in particular to a model search method, device, equipment, storage medium, and program product. Background Art

[0002] Model search refers to the process of searching for the most suitable network model from the hypernetwork for image processing tasks to improve the processing effect of image processing tasks. Model search can be widely used in scenarios such as image segmentation and image recognition.

[0003] The model search method in the related art is to search for operation operators, which results in structural limitations in the searched network model, making it more difficult to search for the optimal network model, thus affecting the performance of the model search. Summary of the invention

[0004] The embodiments of the present application provide a model search method, apparatus, device, computer-readable storage medium, and program product, which can improve the performance of model search.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a model search method, including:

[0007] Based on the search data set, a first weight of a network granularity and multiple second weights of an operator granularity are searched for multiple network units in a target super network from a search space; wherein the network granularity is a weight granularity that affects an external structure of a search model, the operator granularity is a weight granularity that affects an operator inside the search model, and the target super network is a set consisting of all candidate network structures;

[0008] Extracting at least two target network units from a plurality of network units of the target super network according to the first weight, and obtaining a target network structure by connecting the at least two target network units;

[0009] Based on the plurality of the second weights and the plurality of operation operators, generating a target operator corresponding to each of the target network units of the target network structure;

[0010] The target operator and each of the target network units are used to construct a target network model and complete the model search.

[0011] The present application provides a model search device, including:

[0012] A weight search module, for searching for a first weight of a network granularity and a plurality of second weights of an operator granularity for a plurality of network units in a target super network from a search space based on a search data set; wherein the network granularity is a weight granularity that affects an external structure of a search model, the operator granularity is a weight granularity that affects an operator of the search model, and the target super network is a set consisting of all candidate network structures;

[0013] A structure determination module, configured to extract at least two target network units from a plurality of network units of the target super network according to the first weight, and obtain a target network structure by connecting the at least two target network units;

[0014] An operator generation module, used to generate a target operator corresponding to each of the target network units of the target network structure based on the plurality of the second weights and the plurality of operation operators;

[0015] The model building module is used to construct a target network model using the target operator and each of the target network units to complete the model search.

[0016] In some embodiments of the present application, the weight search module is also used to construct a sparse supernetwork based on the search space; the depth and width of the target supernetwork are greater than the depth and width of the sparse supernetwork; the sparse supernetwork is iteratively updated through the search data set, and a candidate space is determined from the search space; the target supernetwork is iteratively updated through the search data set, and the first weight of the network granularity and multiple second weights of the operator granularity are searched for each network unit of the target supernetwork from the candidate space.

[0017] In some embodiments of the present application, the search data set includes: training image data and verification image data; the weight search module is also used to update the model parameters of the first initial supernetwork of the kth iteration based on the training image data to obtain the first temporary supernetwork of the kth iteration; wherein k is a positive integer, and the first initial supernetwork of the 1st iteration is the sparse supernetwork; based on the verification image data, the weight of the network granularity of each network unit of the first temporary supernetwork of the kth iteration is updated to obtain the first intermediate supernetwork of the kth iteration; based on the verification image data, the weight of the operator granularity of each network unit of the first intermediate supernetwork of the kth iteration is updated to obtain the first updated supernetwork of the kth iteration, and the first updated supernetwork of the kth iteration is used as the first initial supernetwork of the k+1th iteration; when k reaches M, the candidate space is determined from the search space based on the weight of the operator granularity of each network unit of the first updated supernetwork of the Mth iteration; M is the total number of first iterations.

[0018] In some embodiments of the present application, the weight search module is also used to perform difference calculation on the candidate weights of the operator granularity in the search space and the weights of the operator granularity of each network unit of the first updated supernetwork of the Mth iteration to obtain the weight difference; from the search space, the candidate weights of the operator granularity whose weight difference is greater than the difference threshold are eliminated to obtain the candidate space.

[0019] In some embodiments of the present application, the weight search module is also used to perform regional segmentation of the object of interest on the training image data through the first initial supernetwork of the kth iteration to obtain a first segmented region; and use the first segmented region and the loss value between the labeled region of the object of interest in the training image data to update the model parameters of the first initial supernetwork of the kth iteration to obtain a first temporary supernetwork of the kth iteration.

[0020] In some embodiments of the present application, the weight search module is also used to perform regional segmentation of the object of interest on the verification image data through the first temporary supernetwork of the kth iteration to obtain a second segmented area; using the second segmented area and the loss value between the labeled area of ​​the object of interest in the verification image data, the weight of the network granularity of each network unit of the first temporary supernetwork of the kth iteration is updated to obtain the first intermediate supernetwork of the kth iteration.

[0021] In some embodiments of the present application, the weight search module is also used to perform regional segmentation of the object of interest on the verification image data through the first intermediate super network of the kth iteration to obtain a third segmented area; using the third segmented area and the loss value between the labeled area of ​​the object of interest in the verification image data, the weight of the operator granularity of each network unit of the first intermediate super network of the kth iteration is updated to obtain the first updated super network of the kth iteration.

[0022] In some embodiments of the present application, the search data set includes: training image data and verification image data; the weight search module is also used to update the model parameters of the second initial supernetwork of the i-th iteration based on the training image data to obtain the second temporary supernetwork of the i-th iteration; i is a positive integer, and the second initial supernetwork of the 1st iteration is the target supernetwork; based on the verification image data, the weight of the network granularity of each network unit of the second temporary supernetwork of the i-th iteration is updated to obtain the second intermediate supernetwork of the i-th iteration; based on the verification image data, the weight of the operator granularity of each network unit of the second intermediate supernetwork of the i-th iteration is updated to obtain the second updated supernetwork of the i-th iteration, and the second updated supernetwork of the i-th iteration is used as the second initial updated network of the i+1-th iteration; when i reaches N, the weight of the network granularity of each network unit of the second updated supernetwork of the N-th iteration is determined as the first weight, and the weight of the operator granularity of each network unit of the second updated supernetwork of the N-th iteration is determined as the second weight; N is the total number of second iterations.

[0023] In some embodiments of the present application, the model search device also includes: a data set construction module; the data set construction module is used to obtain image data sets in multiple fields before searching for the first weight of the network granularity and the multiple second weights of the operator granularity for the multiple network units in the target super network from the search space based on the search data set; based on a preset probability distribution, determine the corresponding extraction ratio for the image data set in each field; extract the image data to be mixed in each field from the image data set in each field according to the extraction ratio; integrate the image data to be mixed in multiple fields into the search data set.

[0024] In some embodiments of the present application, the operator generation module is further used to fuse the output layers of the plurality of operation operators based on the plurality of the second weights to obtain a target operator corresponding to each of the target network units of the target network structure, wherein the plurality of operation operators at least include: a horizontal fusion operator, an upsampling operator, and a downsampling operator; wherein the horizontal fusion operator is an operator whose input feature size is the same as the output feature size, the upsampling operator is an operator whose input feature size is smaller than the output feature size, and the downsampling operator is an operator whose input feature size is smaller than the output feature size;

[0025] The model building module is also used to add the target operator to each target network unit of the target network structure to obtain the target network model and complete the model search.

[0026] In some embodiments of the present application, the model search device also includes: an image segmentation module; the image segmentation module is used to construct a target network model using the target operator and each of the target network units, and after completing the model search, use the target network model to segment the region of interest where the object of interest is located from the acquired medical image data.

[0027] The present application provides a model search device, including:

[0028] A memory for storing executable instructions;

[0029] The processor is used to implement the model search method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0030] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the model search method provided in the embodiment of the present application.

[0031] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the model search method provided in the embodiment of the present application.

[0032] The embodiments of the present application have the following beneficial effects: the model search device can simultaneously search for each network unit in the target super network to obtain the first weight of the network granularity and the second weight of the operator granularity, and determine the target network unit required by the network model through the first weight to connect to obtain the external network structure, that is, determine the target network structure, and construct the target operator of each network unit through the second weight and different operation operators, thereby realizing network-level search and operator-level search at the same time during model search, so that the structure of the searched network model can be made more diversified, thereby improving the performance of model search. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the structure of the encoding-decoding network;

[0034] Figure 2 This is a schematic diagram of a variant network of an encoding-decoding network generated based on model search;

[0035] Figure 3 It is a schematic diagram of the architecture of the model search system provided in the embodiment of the present application;

[0036] Figure 4 The embodiment of this application provides Figure 3 A schematic diagram of the structure of the server in FIG.

[0037] Figure 5It is a flowchart of the model search method provided in the embodiment of the present application;

[0038] Figure 6 is another flowchart of the model search method provided in an embodiment of the present application;

[0039] Figure 7 It is another flowchart of the model search method provided in the embodiment of the present application;

[0040] Figure 8 is a schematic diagram of a virtual data set provided in an embodiment of the present application;

[0041] Fig. 9 is a schematic diagram of the structure of a braided network provided in an embodiment of the present application;

[0042] Fig.10 is a schematic diagram of a conventional unit provided in an embodiment of the present application;

[0043] Fig.11 This is a comparison chart of the effects of segmenting medical image data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0049] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0050] Artificial intelligence technology is a comprehensive study that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, deep learning / deep learning, intelligent driving and other major directions.

[0051] 2) Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0052] 3) Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0053] 4) Nerral Architecture Search (NAS) is a strategy for automatically designing neural networks. By setting a certain search space, a search strategy is designed to find the network structure that performs best on the validation dataset in the search space.

[0054] 5) Continuous relaxation is a method of making discrete space continuous. In essence, it uses the Softmax function to transform a sequence from discrete space to continuous space.

[0055] 6) Super Network refers to the set of all possible sub-networks in the model search process. Based on the set search space, a super network can be generated, which contains multiple sub-networks. After training, these sub-networks can be used for the performance indicators being evaluated.

[0056] 7) Network unit, used to stack modules (Blocks) to generate network structures.

[0057] 8) Operators: small network modules composed of multiple convolutional layers, pooling layers and other basic network layers, which can be used to operate images (or feature maps), such as feature combination or compression of feature maps. Operators can be divided into downsampling operators, upsampling operators, standard operators (i.e., operators with the same input and output sizes), compression operators, multi-scale operators, etc. according to the operations implemented.

[0058] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of artificial intelligence technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0059] Model search refers to the process of searching for the most suitable network model from the hypernetwork for image processing tasks to improve the processing effect of image processing tasks. Model search can be widely used in scenarios such as image segmentation and image recognition.

[0060] In related technologies, model search is performed on each network unit in the network structure to search for model operators, that is, through model search, suitable operation operators are searched from the search space for the network units in the network structure, and the operation operators and the existing network structure are used to integrate and obtain the final network model.

[0061] However, the model search method in the related art is to search for operation operators, so that the structure of the network model obtained by the search is always restricted by the network structure used for model search, which makes the network model structurally limited.

[0062] For example, Figure 1 : is a schematic diagram of the structure of the encoder-decoder network. The encoder-decoder network 1-1 consists of an encoding path 1-11 and a decoding path 1-12, and there is a skip connection 1-13 between the encoding path 1-11 and the decoding path 1-12. In the encoding path 1-11, the amount of calculation is reduced, the receptive field is increased, and the robustness to small input fluctuations is improved to reduce overfitting; in the decoding path 1-12, the pixel loss in the downsampling process is restored by the upsampling operation to perform the end-to-end image segmentation task. However, the encoder-decoder network 1-1 cannot restore the spatial information loss generated in the downsampling process, so the skip connection 1-13 is introduced to fuse the low-order features on the encoding path (the low-order features have more spatial structure information) and the high-order features on the decoding path (the high-order features have lost spatial structure information), so that the fused features integrate more underlying features, so that the feature map after segmentation has more accurate edge information.

[0063] Figure 2 This is a schematic diagram of a variant network based on model search to generate an encoder-decoder network. Figure 2 The variant network 2-2 searches for a suitable operation operator 2-21 from the standard operator (Normal Cell), the compression operator (Reduce Cell) and the multi-scale operator (Multi-Scale Cell) for each network unit 2-22 in the encoding-decoding network 2-1, thereby making the structure of the network unit 2-22 of the variant network 2-2 more diversified to adapt to image segmentation tasks in different fields.

[0064] It can be seen that the variant network generated based on model search has a great similarity with the structure of the decoding-encoding network, which means that the structure of the searched network model is limited, making it difficult to search for the optimal network model, thus affecting the performance of the model search.

[0065] In addition, most of the data sets used for model search in related technologies are in the field of natural images (e.g., images of life scenes, images of plants and animals, etc.). When model search is applied to the field of image segmentation, especially medical image segmentation, since medical image data is very limited and has a strong domain gap, when performing model search for medical image segmentation tasks, it is more inclined to choose parameter-free operations, which affects the performance of the searched network model, and even makes the performance of model search poor.

[0066] The embodiments of the present application provide a model search method, apparatus, device, computer-readable storage medium, and program product, which can improve the performance of model search. The following describes an exemplary application of the model search device provided in the embodiments of the present application. The model search device provided in the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as a server. The following describes an exemplary application of the model search device implemented as a server.

[0067] See also Figure 3 , Figure 3 : is a schematic diagram of the architecture of the model search system provided in the embodiment of the present application. In order to support a model search application, in the model search system 100, the terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two. In the model search system 100, a database 500 is also provided to provide data support to the server 200. The database 500 can be integrated in the server 200, or it can be independent of the server 200. Figure 1 The illustrated case is that the database 500 is independent of the server 200 .

[0068] The terminal 400 - 1 is used to generate a search data set and transmit the search data set to the server 200 through the network 300 .

[0069] Server 200 is used to search for a first weight of network granularity and multiple second weights of operator granularity for multiple network units in a target super network from a search space based on a search data set; wherein the network granularity is a weight granularity that affects the external structure of a search model, the operator granularity is a weight granularity that affects operators inside the search model, and the target super network is a set consisting of all candidate network structures; according to the first weight, at least two target network units are extracted from multiple network units of the target super network, and a target network structure is obtained by connecting the at least two target network units; based on multiple second weights and multiple operation operators, a target operator corresponding to each target network unit of the target network structure is generated; using the target operator and each target network unit, a target network model is constructed to complete the model search.

[0070] The server 200 is further used to send the target network model to the terminal 400 - 2 . The terminal 400 - 2 is used to segment the region of interest for the medical image data using the target network model and display the region of interest in the graphical interface 410 - 2 .

[0071] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart home appliance, a vehicle-mounted terminal, a medical image analysis device, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0072] See also Figure 4 , Figure 4 The embodiment of this application provides Figure 3 A schematic diagram of the structure of a server (an implementation of a model search device) in FIG. Figure 4 The server 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 4 Various buses are labeled as bus system 240 .

[0073] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0075] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0076] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0077] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0078] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0079] A network communication module 252, used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB);

[0080] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., display screen, speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripherals and displaying content and information);

[0081] The input processing module 254 is used to detect one or more user inputs or interactions from one of the one or more input devices 232 and translate the detected inputs or interactions.

[0082] In some embodiments, the model search device provided in the embodiments of the present application can be implemented in software. Figure 4 The model search device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a weight search module 2551, a structure determination module 2552, an operator generation module 2553, a model construction module 2554, a data set construction module 2555 and an image segmentation module 2556. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0083] In other embodiments, the model search device provided in the embodiments of the present application can be implemented in hardware. As an example, the model search device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model search method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0084] In some embodiments, the server (an implementation of the model search device) can implement the model search method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a model search APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0085] The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, vehicle-mounted, medical, etc. Below, the model search method provided by the embodiments of the present application will be described in combination with the exemplary application and implementation of the model search device provided by the embodiments of the present application.

[0086] See also Figure 5 , Figure 5 is a flow chart of the model search method provided in the embodiment of the present application, which will be combined with Figure 5 The steps shown are explained.

[0087] S101. Based on a search data set, search for a first weight of a network granularity and a plurality of second weights of an operator granularity for a plurality of network units in a target super network from a search space.

[0088] The embodiment of the present application is implemented in the scenario of model search for image processing tasks, for example, model search for medical image segmentation tasks, and model search for object recognition tasks. When the model search process starts, first, the model search device will obtain the search data set from the database or its own storage space, and then input the search data set into the constructed target super network, so as to use the search data set to search the weight of each network unit in the target super network at the network granularity in the search space, and finally determine the first weight of each network unit, and search the weight of each network unit at the operator granularity, and determine the second weight of each of the multiple operation operators for each network unit, thereby obtaining multiple second weights.

[0089] It should be noted that the network granularity is the weight granularity that affects the external structure of the search model, the operator granularity is the weight granularity that affects the operators inside the search model, and the target super network is a set of all candidate network structures. That is, from the target super network, multiple different network models can be created based on the weight of the network granularity and the weight of the operator granularity of each network unit. The first weight is the weight of the searched better network granularity, and the second weight is the weight of the searched better operator granularity. The model search device affects the external structure of the search model (for example, whether there are skip connections between different network units, how many network units the model has, etc.) according to the first weight of each network unit, and affects the operators inside the search model (for example, whether a certain network unit is an upsampling operator or a downsampling operator) according to the multiple second weights of each network unit. In this way, a more reasonable network model can be created for the image processing task.

[0090] In some embodiments, the model search device can update and iterate the target super network by searching the data set to obtain the first weight and the second weight from the search space, that is, the first weight and the second weight are obtained through one search.

[0091] In other embodiments, the model search device may first construct a supernetwork that is smaller than the target supernetwork, and then iterate the supernetwork by searching the data set, determine a smaller candidate space from the search space, and use the search data set to iterate the target supernetwork, search for the first weight and the second weight from the candidate search space, thereby obtaining the first weight and the second weight through two stages of searching.

[0092] It can be understood that the search space has candidate weights that can be selected at the network granularity for each network unit of the super network, and candidate weights that can be selected at the operator granularity for each unit.

[0093] S102: extract at least two target network units from multiple network units of the target super network according to the first weight, and obtain a target network structure by connecting the at least two target network units.

[0094] The model search device uses the first weight obtained by the search to determine whether each network unit in the target super network can bring an improvement in processing performance to the image processing task (for example, improved accuracy, faster reasoning speed, etc.), so as to extract the network unit that brings improved processing performance for the image processing task from the multiple network units of the target super network, and use the extracted network unit as the target network unit. In this way, the model search device can obtain at least two target network units. Then, the model search device will connect at least two target network units in the order of the target network units in the target super network, or in the opposite order of the target network units in the target super network, to obtain the target network structure.

[0095] In some embodiments, the model search device may determine the network unit whose first weight is greater than the weight threshold as the target network unit. In other embodiments, the model search device may also determine the N network units with the largest first weights as the target network units. The embodiments of the present application are not specifically limited here.

[0096] It should be noted that the target network unit determined in this step has not yet added operation operators. Therefore, the target network structure spliced ​​together by multiple target network units only represents the external network structure of the searched model, such as the number of network units, the position of each network unit in the target structure model, etc.

[0097] S103: Generate a target operator corresponding to each target network unit of the target network structure based on the multiple second weights and the multiple operation operators.

[0098] In some embodiments, the model search device may weight multiple operation operators according to multiple second weights, and use the weighted operation operators for fusion to obtain a target operator for each target network unit.

[0099] In other embodiments, the model search device may compare multiple second weights with the weight threshold of the operator granularity, and then extract the operation operator whose second weight is greater than or equal to the weight threshold from the multiple operation operators, and determine the extracted operation operator as the target operator.

[0100] S104: Utilize the target operator and each target network unit to construct a target network model and complete the model search.

[0101] After the model search device obtains the target operator corresponding to each target network unit through fusion, the target operator can be added to each target network unit so that each target network unit can process the image (or feature map) to obtain a target network model that can be used for image processing tasks. Pruning, compression and other processing can also be performed on some target operators, and the processed target operators can be added to the corresponding target network units to obtain the target network model. The embodiments of the present application are not limited here.

[0102] It can be understood that compared with the model search in the related art, which searches for operation operators and limits the structure of the network model obtained by the search, in the embodiment of the present application, the model search device can simultaneously search for each network unit in the target super network to obtain the first weight of the network granularity and the second weight of the operator granularity, and determine the target network unit required by the network model through the first weight to connect to obtain the external network structure, that is, determine the target network structure, and construct the target operator of each network unit through the second weight and different operation operators, thereby realizing network-level search and operator-level search at the same time during the model search. In this way, the structure of the network model obtained by the search can be more diversified, thereby improving the performance of the model search.

[0103] based on Figure 5 , see Figure 6 , Figure 6 It is another flow chart of the model search method provided by the embodiment of the present application. In some embodiments of the present application, based on the search data set, the first weight of the network granularity and the multiple second weights of the operator granularity are searched for the multiple network units in the target super network from the search space, that is, the specific implementation process of S101, which may include: S1011-S1013, as follows:

[0104] S1011. Construct a sparse hypernetwork based on the search space.

[0105] S1012. Iteratively update the sparse hypernetwork by searching the data set, and determine a candidate space from the search space.

[0106] It should be noted that the depth and width of the target supernetwork are greater than the depth and width of the sparse supernetwork. That is to say, the model search device reconstructs a smaller-scale sparse supernetwork for the search space, and then initializes the network granularity weights and operator granularity weights of each network unit of the sparse supernetwork from the search space, and uses the search data set to update and iterate the network granularity weights and operator granularity weights obtained by initializing each network unit of the sparse supernetwork, so as to screen the candidate weights of the network granularity and the candidate weights of the operator granularity in the search space with good reasoning effect on the sparse supernetwork on the search data set (which can be determined by the weights of the network granularity and the weights of the operator granularity selected for the sparse supernetwork after the update iteration is completed), and eliminates the candidate weights of the network granularity and the candidate weights of the operator granularity in the search space with poor reasoning effect on the sparse supernetwork on the search data set, that is, extracting a better candidate space from the search space.

[0107] S1013, iteratively updating the target super network by searching the data set, and searching for a first weight of the network granularity and multiple second weights of the operator granularity for each network unit of the target super network from the candidate space.

[0108] After the model search device determines the candidate space from the search space, it initializes the weights of the network granularity and the operator granularity for each network unit of the target super network from the candidate space, and then uses the search data set to iterate the target super network to continuously update the weights of the network granularity of each network unit in the target super network until the update iteration is completed, and the weights of the network granularity after the last round of update are used as the first weights, and multiple second weights are obtained for each network unit in the same way.

[0109] It can be understood that in the embodiments of the present application, since the scale of the sparse hypernetwork is small, the parameters that need to be calculated are also small, and the time required for iterative updating is less. When the model search device performs the first stage search based on the sparse hypernetwork, it is possible to determine the candidate space from the search space in a shorter time, that is, to effectively reduce the scale of the search space in a shorter time, so that the first weight and the second weight can be directly searched from the smaller candidate space for the target hypernetwork, thereby effectively improving the efficiency of the model search.

[0110] In some embodiments of the present application, the search data set includes: training image data and verification image data. The sparse hypernetwork is iteratively updated through the search data set to determine the candidate space from the search space, that is, the specific implementation process of S1012, which may include: S1012a-S1012d, as follows:

[0111] S1012a. Based on the training image data, the model parameters of the first initial hypernetwork of the k-th iteration are updated to obtain the first temporary hypernetwork of the k-th iteration, wherein the initial hypernetwork of the first iteration is a sparse hypernetwork.

[0112] If a positive integer k is used to represent the update iteration round for the sparse hypernetwork, then, in the kth iteration, the model search device will first use the training image data in the search data set to update the model parameters of the first initial hypernetwork of the kth iteration, and determine the hypernetwork obtained after the model parameter update is completed as the temporary hypernetwork of the kth iteration. Wherein, k is a positive integer,

[0113] It should be noted that the model search device uses the constructed sparse hypernetwork as the first initial hypernetwork of the first round of iteration, and uses the hypernetwork obtained after the first round of iteration as the first initial hypernetwork of the next round of iteration. This cycle is repeated to achieve iterative update of the sparse hypernetwork.

[0114] It can be understood that for any hypernetwork, the model parameters are the parameters at the lowest level, which can be understood as the parameters of the network layers that constitute the operation operators, such as the parameters of the convolutional layer, the pooling layer, etc.

[0115] S1012b. Based on the verification image data, update the weight of the network granularity of each network unit of the first temporary super network of the k-th iteration to obtain the first intermediate super network of the k-th iteration.

[0116] After obtaining the first temporary supernetwork of the k-th iteration, the model search device will input the verification image data into the first temporary supernetwork of the k-th iteration, so as to update the weight of the network granularity of each network unit of the first temporary supernetwork of the k-th iteration based on the inference result in the verification image data and the difference in the label of the verification image data through the first temporary supernetwork of the k-th iteration. The supernetwork obtained after the weight update of the network granularity is completed is the first intermediate supernetwork of the k-th iteration.

[0117] It should be noted that the training image data and verification image data in the search data set both have corresponding labels, wherein the label of the training image data indicates the annotated area of ​​the target object in the training image data, and the label of the verification image data indicates the annotated area of ​​the target object in the verification image data.

[0118] S1012c. Based on the verification image data, the weight of the operator granularity of each network unit of the first intermediate supernetwork of the k-th iteration is updated to obtain the first updated supernetwork of the k-th iteration, and the first updated supernetwork of the k-th iteration is used as the first initial supernetwork of the k+1-th iteration.

[0119] After obtaining the first intermediate super-network of the k-th iteration, the model search device will input the verification image data into the first intermediate super-network of the k-th iteration again, so as to update the weight of the operator granularity of each network unit of the first intermediate super-network of the k-th iteration based on the difference between the inference result of the verification image data and the label of the verification image data by the first intermediate super-network of the k-th iteration, and determine the super-network obtained after the update as the first updated super-network of the k-th iteration, and finally use the first updated super-network of the k-th iteration as the starting point of the k+1-th iteration to continue the model parameter update process of the k+1-th iteration.

[0120] S1012d: When k reaches M, determine a candidate space from the search space based on the weight of the operator granularity of each network unit of the first updated supernetwork in the Mth iteration.

[0121] It should be noted that M is the total number of first iterations. When k reaches M, the model search device enters the last round of iterations for the sparse hypernetwork, i.e., the Mth round of iterations, to obtain the first updated hypernetwork of the Mth round of iterations. Next, the model search device uses the weights of the operator granularity of each network unit of the first updated hypernetwork of the Mth round of iterations to eliminate the weights of the operator granularity with poor performance in the search space, thereby obtaining a candidate space.

[0122] In an embodiment of the present application, the model search device can update the model parameters, network granularity weights and operator granularity weights of the supernetwork in the kth iteration by searching the training image data and verification image data in the data set, and use the weights of the operator granularity of the last round to narrow the search space, thereby reducing the number of parameters in the search space.

[0123] In some embodiments of the present application, based on the weight of the operator granularity of each network unit of the first updated super network of the Mth iteration, the candidate space is determined from the search space, that is, the specific implementation process of S1012d may include: S201-S202, as follows:

[0124] S201 , performing difference calculation on the candidate weights of the operator granularity in the search space and the weights of the operator granularity of each network unit of the first updated supernetwork of the Mth iteration to obtain the weight difference.

[0125] It can be understood that the model search device can obtain the weight difference by subtracting the candidate weight of the operator granularity in the search space from the weight of the operator granularity of each network unit of the first updated supernetwork in the Mth round of iteration, or it can obtain the weight difference by comparing the candidate weight of the operator granularity with the weight of the operator granularity of each network unit of the first updated supernetwork in the Mth round of iteration. The embodiments of the present application are not limited here.

[0126] S202 : Eliminate candidate weights of operator granularity whose weight difference is greater than a difference threshold from the search space to obtain a candidate space.

[0127] The model search device obtains a difference threshold, then compares the weight difference with the difference threshold, and removes the candidate weights of the operator granularity corresponding to the weight difference greater than the difference threshold from the search space, and determines the search space that completes the removal operation as the candidate space.

[0128] It is understandable that the difference weight can be set manually, or it can be determined by the model search device with the help of artificial intelligence technology after analyzing the domain of the search data set or the similarity of each image data in the search data set. The embodiments of the present application are not limited here.

[0129] In some embodiments of the present application, based on the training image data, the model parameters of each network unit of the first initial super network of the k-th iteration are updated to obtain the first temporary super network of the k-th iteration, that is, the specific implementation process of S1012a may include: S203-S204, as follows:

[0130] S203 . Perform region segmentation of the object of interest on the training image data through the first initial hypernetwork of the kth iteration to obtain a first segmented region.

[0131] The model search device inputs the training image data into the first initial network of the k-th iteration, so as to segment the area where the object of interest is located from the training image data using the first initial network of the k-th iteration to obtain a first segmented area.

[0132] It is understandable that the object of interest may be a lesion, an organ, etc. in the medical field, or may be a common object in daily life, such as a tree, a roadside traffic sign, and so on.

[0133] S204, using the first segmented area and the loss value between the labeled area of ​​the object of interest in the training image data, update the model parameters of the first initial hypernetwork of the k-th iteration to obtain the first temporary hypernetwork of the k-th iteration.

[0134] The model search device calculates the loss for the first segmented area and the annotated area where the object of interest is actually located in the training image data, obtains the loss value of the first segmented area and the annotated area corresponding to the training image data, and then solves the partial derivative of the loss value for the original model parameters in the first initial supernetwork of the kth iteration, and then uses the product between the partial derivative and the learning rate to calculate the update component of the model parameters, and uses the update component to update the model parameters of the first initial supernetwork of the kth iteration. After the update is completed, the first temporary supernetwork of the kth iteration can be obtained.

[0135] Exemplarily, formula (1) is a calculation process for updating the model parameters of the first initial hypernetwork in the kth iteration:

[0136]

[0137] Among them, w k is the original model parameter of the first initial hypernetwork in the kth iteration, η w represents the learning rate of the model parameters, represents the training image data in the search dataset, α k is the weight of the operator granularity of each network unit of the first initial supernetwork in the kth iteration, β k is the weight of the network granularity of each network unit of the first initial supernetwork of the kth iteration, is the loss value of the first segmented area and the labeled area corresponding to the training image data, is the partial derivative operation, w k+1 is the updated model parameter of the first initial super network of the kth iteration. According to formula (1), the model search device can complete the update of the model parameter of the first initial super network of the kth iteration.

[0138] In some embodiments of the present application, based on the verification image data, the weight of the network granularity of each network unit of the first temporary super network of the kth iteration is updated to obtain the first intermediate super network of the kth iteration, that is, the specific implementation process of S1012b may include: S205-S206, as follows:

[0139] S205 . Segment the verification image data into regions of interest objects using the first temporary hypernetwork of the kth iteration to obtain second segmented regions.

[0140] S206. Using the second segmented area and the loss value between the labeled areas of the object of interest in the verification image data, the weight of the network granularity of each network unit of the first temporary supernetwork of the kth iteration is updated to obtain the first intermediate supernetwork of the kth iteration.

[0141] The model search device inputs the verification image data into the first temporary super network of the kth iteration, and uses the first temporary super network of the kth iteration to segment the area where the object of interest is located from the verification image data, and determines the area as the second segmented area. Then, the model search device calculates the loss value of the second segmented area and the area where the object of interest is located in the verification image data, that is, the loss value of the annotated area corresponding to the verification image data, and obtains the partial derivative of the weight of the network granularity for the loss value, such as the learning rate of the solved partial derivative value and the weight of the network granularity, determines the update component of the weight of the network granularity, and then uses the determined update component to update the weight of the network granularity of each network unit of the first temporary super network of the kth iteration, and after the update is completed, the first intermediate super network of the kth iteration is obtained.

[0142] The process of updating the weight of the network granularity of each network unit of the first temporary super network of the kth iteration in the embodiment of the present application can be expressed as formula (2):

[0143]

[0144] Among them, β k is the weight of the network granularity of each network unit of the first temporary supernetwork in the kth iteration, w k+1 is the model parameter of the first temporary hypernetwork of the kth iteration (that is, the updated model parameter of the first initial hypernetwork of the kth iteration), α k is the weight of the operator granularity of each network unit of the first initial supernetwork in the kth iteration, (w k+1 ; α k ,β k ) is the first temporary super network of the kth iteration, is the verification image data, is the loss of the second segmented area and the annotated area corresponding to the verification image data, η β is the learning rate corresponding to the weight of the network granularity, β k+1 is the weight of the updated network granularity of each network unit in the first temporary supernetwork of the kth iteration, According to formula (2), the model search device can complete the update of the weights of the network units of the first intermediate super network in the kth iteration at the network granularity.

[0145] In some embodiments of the present application, based on the verification image data, the weight of the operator granularity of each network unit of the first intermediate super network of the kth iteration is updated to obtain the first updated super network of the kth iteration, that is, the specific implementation process of S1012c may include: S207-S208, as follows:

[0146] S207, using the first updated hypernetwork of the kth iteration, segmenting the verification image data into regions of interest, obtaining a third segmented region

[0147] S208. Using the third segmented area and the loss value between the labeled areas of the object of interest in the verification image data, the weight of the operator granularity of each network unit of the first intermediate supernetwork of the kth iteration is updated to obtain the first updated supernetwork of the kth iteration.

[0148] The model search device calculates the loss value of the third segmented area and the annotated area corresponding to the verification image data, and obtains the partial derivative of the weight of the operator granularity for the loss value, such as the solved partial derivative value and the learning rate of the weight of the operator granularity, determines the update component of the weight of the operator granularity, and then uses the determined update component to update the weight of the operator granularity of each network unit of the first intermediate supernetwork of the kth iteration, and determines the supernetwork obtained after the update is completed as the first updated supernetwork of the kth iteration.

[0149] The process of updating the weight of the operator granularity of each network unit of the first updated supernetwork of the kth iteration in the embodiment of the present application can be expressed as formula (3):

[0150]

[0151] Among them, α k is the weight of the operator granularity of each network unit of the first intermediate supernetwork in the kth iteration, β k+1 is the weight of the network granularity of each network unit of the first intermediate super network of the kth iteration (that is, the weight of the updated network granularity of each network unit of the first temporary super network of the kth iteration), (w k+1 ; α k ,β k+1 ) represents the first intermediate super network of the kth iteration, is the loss value of the third segmented area and the annotated area corresponding to the verification image data, η α is the learning rate corresponding to the weight of the operator granularity, is the partial derivative operation, α k+1 is the weight of each network unit of the first intermediate super network in the kth iteration at the operator granularity after the update. According to formula (3), the model search device can complete the update of the weight of the network unit of the first intermediate super network in the kth iteration at the operator granularity.

[0152] In some embodiments of the present application, when the search data set includes: training image data and verification image data, the target super network is iteratively updated by searching the data set, and the first weight of the network granularity and the multiple second weights of the operator granularity are searched for each network unit of the target super network from the candidate space, that is, the specific implementation process of S1013 may include: S1013a-S1013d, as follows:

[0153] S1013a. Based on the training image data, the model parameters of the second initial hypernetwork of the i-th iteration are updated to obtain the second temporary hypernetwork of the i-th iteration.

[0154] Wherein, i is a positive integer, and the second initial supernetwork of the first iteration is the target supernetwork.

[0155] S1013b, based on the verification image data, update the weight of the network granularity of each network unit of the second temporary super network of the i-th iteration to obtain the second intermediate super network of the i-th iteration

[0156] S1013c. Based on the verification image data, the weight of the operator granularity of each network unit of the second intermediate supernetwork of the i-th iteration is updated to obtain the second updated supernetwork of the i-th iteration, and the second updated supernetwork of the i-th iteration is used as the second initial supernetwork of the i+1-th iteration.

[0157] It should be noted that the specific implementation process of S1013a-S1013c is similar to the specific implementation process of S1012a-S1012c, and will not be repeated here.

[0158] S1013d. When i reaches N, the weight of the network granularity of each network unit of the second updated super network of the Nth iteration is determined as the first weight, and the weight of the operator granularity of each network unit of the second updated super network of the Nth iteration is determined as the second weight.

[0159] Wherein, N is the total number of second iterations. When i reaches N, that is, the total number of second iterations, the model search device will perform a final round of iterative updates on the target supernetwork to obtain the second updated supernetwork of the Nth iteration. Afterwards, the model search device will extract the weight of each network unit of the second updated supernetwork of the Nth iteration at the network granularity to obtain the first weight, and extract the weight of each network unit at the operator granularity to obtain multiple second weights. At this point, the model search device has completed the process of determining the first weight and the second weight.

[0160] based on Figure 6 , see Figure 7 , Figure 7It is another flow chart of the model search method provided by the embodiment of the present application. In some embodiments of the present application, based on the search data set, before searching for the first weight of the network granularity and the multiple second weights of the operator granularity for the multiple network units in the target super network from the search space, that is, before S101, the method may also include: S105-S108, as follows:

[0161] S105: Obtain image datasets in multiple fields.

[0162] S106 . Based on a preset probability distribution, determine a corresponding extraction ratio for the image data set in each field.

[0163] The model search device can obtain image data sets in multiple different fields from a database or the Internet, such as data sets in the field of medical image segmentation and data sets in the field of medical image classification, and then use a preset probability distribution to allocate the corresponding extraction ratio for the image data sets in each field.

[0164] It is understandable that the preset probability distribution can be a Beta distribution or a Gaussian distribution, and the embodiments of the present application are not limited thereto.

[0165] For example, when the preset probability distribution is Beta distribution, that is, Beta(μ,μ), the model search device can combine the hyperparameter μ∈(0,+∞) to determine {λ1,λ2,…,λ a}, and then for {λ1,λ2,…,λ a} is normalized to obtain the extraction ratio of each field, that is, Here, a is the number of fields.

[0166] S107 , extracting the image data to be mixed in each field from the image data set in each field according to the extraction ratio.

[0167] S108, integrating the data sets to be mixed in multiple fields into a search data set.

[0168] The model search device uses the total number of image data contained in the image data set of each field and the extraction ratio of each field to calculate the number of image data that need to be extracted from the image data set of each field, and then randomly or sequentially extracts the image data to be mixed from the image data set of each field based on this number, thereby obtaining the image data to be mixed in each field, and integrating the image data to be mixed in each field into an image data set, which is the search data set.

[0169] For example, when the extraction ratios of a fields are When , the model search device can obtain the search dataset according to formula (4) for the image dataset of each field:

[0170]

[0171] in, is the search dataset and its labels, {(x1,y1),(x2,y2),…,(x a ,y a )} are image data and corresponding labels extracted from image datasets in multiple fields.

[0172] In an embodiment of the present application, the model search device can mix image data from image data sets in multiple different fields to obtain a search data set, so that the search data set can overcome the domain gaps between different fields, thereby improving the feature generalization ability during model search, helping the model search to find the optimal model with sufficient feature aggregation, and further improving the performance of the model search.

[0173] In some embodiments of the present application, based on multiple second weights and multiple operation operators, a target operator corresponding to each target network unit of the target network structure is generated, that is, a specific implementation process of S103 may include: S1031, as follows:

[0174] S1031. Based on the multiple second weights, the output layers of the multiple operation operators are merged to obtain the target operator corresponding to each target network unit of the target network structure.

[0175] The model search device uses the second weight corresponding to each operator to weight the output layer of each operator, and uses the weighted output layer to fuse into one (it can also be understood that the weighted output layer is connected to the output layer of the next network unit), and obtains the target operator corresponding to each target network unit. That is to say, in this step, the model search device fuses multiple operators into a target operator, rather than selecting an operator from multiple operators as the target operator corresponding to each target network unit. In this way, the target operator corresponding to each target network unit can process the input image (or feature map) sufficiently and enhance the feature generalization ability of the target operator.

[0176] In some embodiments of the present application, the multiple operation operators include at least: a horizontal fusion operator, an upsampling operator, and a downsampling operator. Among them, the horizontal fusion operator is an operator whose input feature size is the same as the output feature size, the upsampling operator is an operator whose input feature size is smaller than the output feature size, and the downsampling operator is an operator whose input feature size is smaller than the output feature size. The horizontal fusion operator, the upsampling operator, and the downsampling operator can all be modeled as a directed acyclic graph (DAG).

[0177] In some embodiments of the present application, a target network model is constructed using a target operator and each target network unit to complete the model search, that is, the specific implementation process of S104 may include:

[0178] S1041. Add the target operator to each target network unit of the target network structure to obtain the target network model, thereby completing the model search.

[0179] The model search device can use the input of the target operator as the input of the target network structure and the output of the target operator as, in this way, it is possible to add the target operator to the corresponding target network unit to obtain a target network model that can process the input image (or feature map).

[0180] In some embodiments of the present application, a target network model is constructed using the target operator and each target network unit to complete the model search, that is, after S104, the method may further include: S109, as follows:

[0181] S109. Using the target network model, segment the region of interest where the object of interest is located from the acquired medical image data.

[0182] That is, after searching for the target network model, the model search device can use the acquired medical image data to perform regional segmentation of the object of interest in the medical image data, so as to mark the region of interest from the medical image data, so as to assist medical personnel in the diagnosis process through the region of interest.

[0183] Of course, the model search device can also send the target network model to professional electronic medical equipment so that the electronic medical equipment can use the target network model to perform image segmentation tasks for medical image data information.

[0184] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0185] The embodiment of the present application is implemented in a scenario where the server performs model search for an image segmentation task in medical image data, so that the searched model can be used to mark the area of ​​interest from the medical image data, such as the area where the lesion is located, the area where the organ is located, and so on.

[0186] The server (model search device) will first build a virtual dataset (search dataset) for the model search process. The virtual dataset mixes medical image data from multiple datasets, which can solve the generalization problem of image segmentation tasks for medical image data during model search.

[0187] The virtual data set can be constructed by formula (4). Figure 8 Schematic diagram of a virtual data set provided in an embodiment of the present application. The server will first determine the image extraction ratio for each of the medical data set 8-1, the medical data set 8-2, and the medical data set 8-3 (image data sets in multiple fields), and then randomly extract image data from the above three medical data sets according to the extraction ratio, and use the extracted image data to form a virtual data set 8-4, so as to facilitate the subsequent use of the virtual data set 8-4 to search for a medical image segmentation model (target network model) from the searchable woven network 8-5 (super network).

[0188] The searchable weaving network consists of upsampling units (upsampling operators), downsampling units (downsampling operators), and regular units (horizontal fusion operators), representing feature fusion in top-down, bottom-up, and horizontal directions, respectively.

[0189] For example, Fig. 9 It is a schematic diagram of the structure of the braided network provided in an embodiment of the present application. Fig. 9 The depth of the braided network in is 6 (i.e., 0 to 5), the width is 10 (i.e., 0 to 9), and the feature map of each network unit (i.e. Fig. 9 The hexagons in the figure are obtained by the weighted sum of the outputs of the up-sampling unit 9-1, the down-sampling unit 9-2 and the conventional unit 9-3. For example, Fig. 9 Features in Figure X 2,8 , by the 9-2 (input is feature Figure X 1,7 ) output, up-sampling unit 9-1 (input is feature Figure X 3,7 ), conventional unit 9-3 (input is feature Figure X 2,6 and X 2,4 ) is weighted by the output of Figure 9-4 It is the output of the image preprocessing network unit (such as the STEM unit).

[0190] The server will search the search space at two levels, unit level (operator granularity) and network level (network granularity), to obtain the optimal combination of different network units and the operations corresponding to each network unit. In the search space, there are all network-level weights and unit-level optional operations (regular units take normal operations (normal-ops), downsampling units take downsampling operations (down-ops) in addition to normal operations, and upsampling units take upsampling operations (up-ops) in addition to normal operations). Taking regular units as an example, they can be modeled as directed acyclic graphs. Exemplarily, Fig.10 is a schematic diagram of a conventional unit provided in an embodiment of the present application, from Fig.10It can be seen that the conventional unit 10-1 can be connected into a directed acyclic graph by the outputs 10-11 of other units and the network layer 10-12 inside it.

[0191] The optimization function of the server when searching for the weaving network can be shown as formula (5):

[0192]

[0193] Among them, α is the mixed weight of the unit-level operation (the weight of the operator granularity), β is the mixed weight of the network level (the weight of the network granularity), and w is the network weight (model parameter). The loss function is and It can be calculated based on several depth maps, for example, Fig. 9 X 1,5 , X 1,7 , X 1,9 They are calculated to be determined by the network weight w and the structural parameters (α and β) respectively. Therefore, the optimization of formula (5) is a three-level optimization problem, where the optimization parameter α is the top optimization variable, β is the middle optimization variable, and w * is the lowest-level optimization variable. Therefore, before optimizing α and β, it is necessary to solve w according to the above constraints. * .

[0194] To make the search process more efficient, the server first uses the complete search space, sets a shallower weaving network (sparse network) with a depth of 5 and a width of 8, searches for a certain number of epochs, and then halves the search space based on the parameters of the unit-level operations after the first phase of the search. This means that the operation sets with poor performance are eliminated. Then, a weaving network with a depth of 6 and a width of 10 is constructed, and the final search is performed in the halved search space (candidate space). This search can balance the search space and efficiency.

[0195] Next, the server search process is described.

[0196] First, obtain a virtual dataset and divide it into a training set (training image data) and a validation set (validation graph data), and set the learning rate at the unit level, the network level, and the network weights.

[0197] In the first search phase, the super network is initialized from the operation set and the shallower braided network, and then the network weights of the super network are iteratively trained using the training set. After a certain number of iterations, a super network with better network weights is obtained, and then the super network with better network weights is used as the starting network for iteration, and the virtual data set is used for iteration. At this time, the server will use the training set to calculate the process of formula (1) to update the network weights, use the verification set to update the network-level mixed weights according to formula (2), and use the verification set to update the unit-level mixed weights according to formula (3) until the first search phase is completed.

[0198] After the first search phase is completed, the server will halve the operation set, that is, for example, the operation set of the conventional unit is O1, the operation set of the upsampling unit is O2, and the operation set of the downsampling unit is O3. The server will start the second search phase based on halve(O1), halve(O2) and halve(O3).

[0199] In the second search phase, the server initializes a supernetwork (target supernetwork) from the halved operation set and the deeper braided network, and then completes the second search phase in a similar manner to the first search phase.

[0200] Next, the performance of the model search provided by the embodiments of the present application is described.

[0201] The server runs the models searched by the model search method provided in the embodiment of the present application and the manually designed models on the three datasets of ISIC2018, CVC and CHAOS-CT respectively. The running results are shown in Table 1:

[0202] Table 1

[0203]

[0204] It can be seen that the models searched by the model search method of the embodiment of the present application are obtained by searching the ISIC data set, the CVC data set, and the CHAOS-CT data set. Except for the model based on CVC, all of them can achieve higher performance than the manual design, and the number of parameters is also less. This is mainly because the CHAOS-CT data set is small, and it is more inclined to choose parameter-free operations during model search, which limits the performance of the searched model. On the basis of the model search method of the embodiment of the present application, the virtual data set is continued to be superimposed, that is, the model obtained based on the virtual data set (models with μ=0, μ=0.5 and μ=1) has better consistency and similarity, so it can be seen that the virtual data set can improve the feature generalization ability of the model search.

[0205] See also Fig.11 , Fig.11It is a comparison chart of the effects of segmenting medical image data provided by the embodiment of the present application. Among them, model 11-1, model 11-2 and model 11-3 are respectively models obtained by searching on virtual data sets based on the model search method provided by the embodiment of the present application, and model 11-4 is an artificially designed model. For image 11-5 of the CVC data set, image 11-6 of the ISIC data set and image 11-7 of the CHAOS-CT data set, model 11-1, model 11-2 and model 11-3 can all obtain smoother and clearer segmentation results than model 11-4, and are closer to the true value 11-8. It can be seen that by using the model search method provided by the embodiment of the present application on a virtual data set, a model with good segmentation effect on medical image data can be searched out, thereby improving the performance of the model search.

[0206] It is understandable that in the embodiments of the present application, user information, such as medical image data and other related data, is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0207] The following is a description of an exemplary structure of the model search device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 4 As shown, the software modules stored in the model search device 255 of the memory 250 may include:

[0208] A weight search module 2551 is used to search for a first weight of a network granularity and a plurality of second weights of an operator granularity for a plurality of network units in a target super network from a search space based on a search data set; wherein the network granularity is a weight granularity that affects an external structure of a search model, the operator granularity is a weight granularity that affects an operator inside the search model, and the target super network is a set consisting of all candidate network structures;

[0209] A structure determination module 2552 is used to extract at least two target network units from multiple network units of the target super network according to the first weight, and obtain a target network structure by connecting the at least two target network units;

[0210] An operator generation module 2553 is used to generate a target operator corresponding to each target network unit of the target network structure based on the plurality of second weights and the plurality of operation operators;

[0211] The model building module 2554 is used to use the target operator and each of the target network units to build a target network model and complete the model search.

[0212] In some embodiments of the present application, the weight search module 2551 is also used to construct a sparse supernetwork based on the search space; the depth and width of the target supernetwork are greater than the depth and width of the sparse supernetwork; the sparse supernetwork is iteratively updated through the search data set, and a candidate space is determined from the search space; the target supernetwork is iteratively updated through the search data set, and the first weight of the network granularity and multiple second weights of the operator granularity are searched for each network unit of the target supernetwork from the candidate space.

[0213] In some embodiments of the present application, the search data set includes: training image data and verification image data; the weight search module 2551 is also used to update the model parameters of the first initial supernetwork of the kth iteration based on the training image data to obtain the first temporary supernetwork of the kth iteration; wherein k is a positive integer, and the first initial supernetwork of the 1st iteration is the sparse supernetwork; based on the verification image data, the weight of the network granularity of each network unit of the first temporary supernetwork of the kth iteration is updated to obtain the first intermediate supernetwork of the kth iteration; based on the verification image data, the weight of the operator granularity of each network unit of the first intermediate supernetwork of the kth iteration is updated to obtain the first updated supernetwork of the kth iteration, and the first updated supernetwork of the kth iteration is used as the first initial supernetwork of the k+1th iteration; when k reaches M, the candidate space is determined from the search space based on the weight of the operator granularity of each network unit of the first updated supernetwork of the Mth iteration; M is the total number of first iterations.

[0214] In some embodiments of the present application, the weight search module 2551 is also used to perform difference calculation on the candidate weights of the operator granularity in the search space and the weights of the operator granularity of each network unit of the first updated supernetwork of the Mth iteration to obtain the weight difference; from the search space, the candidate weights of the operator granularity whose weight difference is greater than the difference threshold are eliminated to obtain the candidate space.

[0215] In some embodiments of the present application, the weight search module 2551 is also used to perform regional segmentation of the object of interest on the training image data through the first initial supernetwork of the kth iteration to obtain a first segmented region; and use the first segmented region and the loss value between the labeled region of the object of interest in the training image data to update the model parameters of the first initial supernetwork of the kth iteration to obtain the first temporary supernetwork of the kth iteration.

[0216] In some embodiments of the present application, the weight search module 2551 is also used to perform regional segmentation of the object of interest on the verification image data through the first temporary supernetwork of the kth iteration to obtain a second segmented area; using the second segmented area and the loss value between the labeled area of ​​the object of interest in the verification image data, the weight of the network granularity of each network unit of the first temporary supernetwork of the kth iteration is updated to obtain the first intermediate supernetwork of the kth iteration.

[0217] In some embodiments of the present application, the weight search module 2551 is also used to perform regional segmentation of the object of interest on the verification image data through the first intermediate super network of the kth iteration to obtain a third segmented area; using the third segmented area and the loss value between the labeled area of ​​the object of interest in the verification image data, the weight of the operator granularity of each network unit of the first intermediate super network of the kth iteration is updated to obtain the first updated super network of the kth iteration.

[0218] In some embodiments of the present application, the search data set includes: training image data and verification image data; the weight search module 2551 is also used to update the model parameters of the second initial supernetwork of the i-th iteration based on the training image data to obtain the second temporary supernetwork of the i-th iteration; i is a positive integer, and the second initial supernetwork of the 1st iteration is the target supernetwork; based on the verification image data, the weight of the network granularity of each network unit of the second temporary supernetwork of the i-th iteration is updated to obtain the second intermediate supernetwork of the i-th iteration; based on the verification image data, the weight of the operator granularity of each network unit of the second intermediate supernetwork of the i-th iteration is updated to obtain the second updated supernetwork of the i-th iteration, and the second updated supernetwork of the i-th iteration is used as the second initial updated network of the i+1-th iteration; when i reaches N, the weight of the network granularity of each network unit of the second updated supernetwork of the N-th iteration is determined as the first weight, and the weight of the operator granularity of each network unit of the second updated supernetwork of the N-th iteration is determined as the second weight; N is the total number of second iterations.

[0219] In some embodiments of the present application, the model search device 255 also includes: a data set construction module 2555; the data set construction module 2555 is used to obtain image data sets in multiple fields before searching for the first weight of the network granularity and the multiple second weights of the operator granularity for multiple network units in the target super network from the search space based on the search data set; based on a preset probability distribution, determine the corresponding extraction ratio for the image data set in each field; extract the image data to be mixed in each field from the image data set in each field according to the extraction ratio; integrate the image data to be mixed in multiple fields into the search data set.

[0220] In some embodiments of the present application, the operator generation module 2553 is further used to fuse the output layers of the multiple operation operators based on the multiple second weights to obtain the target operator corresponding to each of the target network units of the target network structure, and the multiple operation operators at least include: a horizontal fusion operator, an upsampling operator, and a downsampling operator; wherein the horizontal fusion operator is an operator whose input feature size is the same as the output feature size, the upsampling operator is an operator whose input feature size is smaller than the output feature size, and the downsampling operator is an operator whose input feature size is smaller than the output feature size;

[0221] The model building module 2554 is also used to add the target operator to each target network unit of the target network structure to obtain the target network model and complete the model search.

[0222] In some embodiments of the present application, the model search device 255 also includes: an image segmentation module 2556; the image segmentation module 2556 is used to construct a target network model using the target operator and each of the target network units, and after completing the model search, use the target network model to segment the region of interest where the object of interest is located from the acquired medical image data.

[0223] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the model search method described above in the embodiment of the present application.

[0224] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the model search method provided by the present application embodiment, for example, Figure 5 The model search method shown.

[0225] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0226] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0227] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0228] As an example, executable instructions may be deployed to be executed on one computing device (an implementation of a model search device), or on multiple computing devices located at one site, or on multiple computing devices distributed at multiple sites and interconnected by a communication network.

[0229] In summary, through the embodiments of the present application, the model search device can simultaneously search for each network unit in the target super network to obtain the first weight of the network granularity and the second weight of the operator granularity, and determine the target network unit required by the network model through the first weight to connect to obtain the external network structure, that is, determine the target network structure, and construct the target operator of each network unit through the second weight and different operation operators, thereby realizing simultaneous network-level search and operator-level search during model search, so that the structure of the searched network model can be more diversified, thereby improving the performance of model search; it can be realized that the candidate space is determined from the search space in a shorter time, that is, the scale of the search space is effectively reduced in a shorter time, so that the first weight and the second weight can be directly searched from the smaller-scale candidate space for the target super network, effectively improving the search efficiency; and it can mix image data from image data sets in multiple different fields to obtain a search data set, so that the search data set can overcome the domain gap between different fields, thereby helping the model search to find the optimal model with sufficient feature aggregation, and further improving the performance of the model search.

[0230] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model search method, characterized in that: The method comprises: Based on a search data set including image data, a sparse super network constructed based on a search space is iteratively updated, and when a first total number of iterations is reached, a difference calculation is performed on a candidate weight of an operator granularity in the search space and a weight of an operator granularity of each network unit of the iteratively obtained first updated super network to obtain a weight difference; Eliminate, from the search space, the candidate weights of the operator granularity whose weight difference is greater than a difference threshold, to obtain a candidate space; The target super network is iteratively updated through a search data set including image data, and a first weight of a network granularity and a plurality of second weights of an operator granularity are respectively searched for each network unit of the target super network from the candidate space; wherein the network granularity is a weight granularity that affects the external structure of the search model, the operator granularity is a weight granularity that affects the operator inside the search model, and the target super network is a set consisting of all candidate network structures; Extracting at least two target network units from a plurality of network units of the target super network according to the first weight, and obtaining a target network structure by connecting the at least two target network units; Based on the plurality of the second weights and the plurality of operation operators, generating a target operator corresponding to each of the target network units of the target network structure; The target operator and each of the target network units are used to construct a target network model for the image processing task, thereby completing the model search.

2. The method according to claim 1, characterized in that The depth and width of the target hypernetwork are greater than the depth and width of the sparse hypernetwork.

3. The method according to claim 1, characterized in that The search data set includes: training image data and verification image data; the iterative updating of the sparse hypernetwork constructed based on the search space includes: Based on the training image data, the model parameters of the first initial hypernetwork of the kth iteration are updated to obtain the first temporary hypernetwork of the kth iteration; wherein k is a positive integer, and the first initial hypernetwork of the first iteration is the sparse hypernetwork; Based on the verification image data, the weight of the network granularity of each network unit of the first temporary super network of the k-th iteration is updated to obtain the first intermediate super network of the k-th iteration; Based on the verification image data, the weight of the operator granularity of each network unit of the first intermediate supernetwork of the kth iteration is updated to obtain the first updated supernetwork of the kth iteration, and the first updated supernetwork of the kth iteration is used as the first initial supernetwork of the k+1th iteration.

4. The method according to claim 3, characterized in that The updating of the model parameters of the first initial hypernetwork of the k-th iteration based on the training image data to obtain the first temporary hypernetwork of the k-th iteration includes: Performing region segmentation of the object of interest on the training image data through the first initial hypernetwork of the kth iteration to obtain a first segmented region; The model parameters of the first initial hypernetwork of the kth iteration are updated by using the first segmented regions and the loss values ​​between the labeled regions of the object of interest in the training image data to obtain the first temporary hypernetwork of the kth iteration.

5. The method according to claim 3, characterized in that: The method of updating the weight of the network granularity of each network unit of the first temporary super network of the k-th iteration based on the verification image data to obtain the first intermediate super network of the k-th iteration includes: By using the first temporary hypernetwork of the kth iteration, the verification image data is segmented into regions of the object of interest to obtain second segmented regions; Using the second segmented area and the loss value between the labeled areas of the object of interest in the verification image data, the weight of the network granularity of each network unit of the first temporary super network of the kth iteration is updated to obtain the first intermediate super network of the kth iteration.

6. The method according to claim 3, characterized in that: The step of updating the weight of the operator granularity of each network unit of the first intermediate super network of the kth iteration based on the verification image data to obtain the first updated super network of the kth iteration includes: Performing region segmentation of the object of interest on the verification image data through the first intermediate super network of the kth iteration to obtain a third segmented region; Using the third segmented area and the loss value between the labeled areas of the object of interest in the verification image data, the weight of the operator granularity of each network unit of the first intermediate supernetwork of the kth iteration is updated to obtain the first updated supernetwork of the kth iteration.

7. The method according to any one of claims 2 to 6, characterized in that: The search data set includes: training image data and verification image data; the method further includes: Based on the training image data, the model parameters of the second initial hypernetwork of the i-th iteration are updated to obtain the second temporary hypernetwork of the i-th iteration; i is a positive integer, and the second initial hypernetwork of the first iteration is the target hypernetwork; Based on the verification image data, the weight of the network granularity of each network unit of the second temporary super network of the i-th iteration is updated to obtain the second intermediate super network of the i-th iteration; Based on the verification image data, the weight of the operator granularity of each network unit of the second intermediate super network of the i-th iteration is updated to obtain the second updated super network of the i-th iteration, and the second updated super network of the i-th iteration is used as the second initial updated network of the i+1-th iteration; When i reaches N, the weight of the network granularity of each network unit of the second updated supernetwork of the Nth iteration is determined as the first weight, and the weight of the operator granularity of each network unit of the second updated supernetwork of the Nth iteration is determined as the second weight; N is the total number of second iterations.

8. The method according to any one of claims 1 to 6, characterized in that: Before iteratively updating the sparse hypernetwork constructed based on the search space, the method further includes: Obtain image datasets from multiple fields; Based on a preset probability distribution, for the image data set in each field, determining a corresponding extraction ratio; According to the extraction ratio, extracting the image data to be mixed in each field from the image data set in each field; The image data to be mixed in multiple fields are integrated into a search data set including image data.

9. The method according to any one of claims 1 to 6, characterized in that: The generating, based on the plurality of the second weights and the plurality of operation operators, a target operator corresponding to each of the target network units of the target network structure comprises: Based on the plurality of the second weights, the output layers of the plurality of the operation operators are fused to obtain a target operator corresponding to each of the target network units of the target network structure, wherein the plurality of operation operators at least include: a horizontal fusion operator, an upsampling operator, and a downsampling operator; The horizontal fusion operator is an operator whose input feature size is the same as the output feature size, the upsampling operator is an operator whose input feature size is smaller than the output feature size, and the downsampling operator is an operator whose input feature size is smaller than the output feature size; The target network model for the image processing task is constructed by using the target operator and each of the target network units to complete the model search, including: The target operator is added to each target network unit of the target network structure to obtain a target network model for the image processing task, thereby completing the model search.

10. The method according to any one of claims 1 to 6, characterized in that: The target network model for the image processing task is constructed by using the target operator and each of the target network units. After completing the model search, the method further includes: The target network model is used to segment the region of interest where the object of interest is located from the acquired medical image data.

11. A model search device, characterized in that: The device comprises: A weight search module is used to iteratively update a sparse supernetwork constructed based on a search space based on a search data set including image data, and when the first total number of iterations is reached, a difference calculation is performed on the candidate weights of the operator granularity in the search space and the weights of the operator granularity of each network unit of the iteratively updated first supernetwork to obtain a weight difference; from the search space, the candidate weights of the operator granularity whose weight difference is greater than a difference threshold are eliminated to obtain a candidate space; the target supernetwork is iteratively updated through a search data set including image data, and a first weight of the network granularity and multiple second weights of the operator granularity are searched for each network unit of the target supernetwork from the candidate space; wherein the network granularity is a weight granularity that affects the external structure of the search model, the operator granularity is a weight granularity that affects the operator inside the search model, and the target supernetwork is a set composed of all candidate network structures; A structure determination module, configured to extract at least two target network units from a plurality of network units of the target super network according to the first weight, and obtain a target network structure by connecting the at least two target network units; An operator generation module, used to generate a target operator corresponding to each of the target network units of the target network structure based on the plurality of the second weights and the plurality of operation operators; The model building module is used to use the target operator and each of the target network units to construct a target network model for image processing tasks and complete model search.

12. A model search device, characterized in that: The model search device comprises: A memory for storing executable instructions; A processor, configured to implement the model search method according to any one of claims 1 to 10 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the model search method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the model search method according to any one of claims 1 to 10 is implemented.

Images