Visual learning method, system, device, and storage medium

Abstract

Embodiments of the present application disclose a visual learning method, system, device and storage medium, wherein the method comprises: providing underlying data corresponding to visual learning by using a system resource layer; calling the underlying data from the system resource layer by using an interface adaptation layer, and providing interface services for an application layer based on the underlying data; and realizing the visual learning based on the interface services provided by the interface adaptation layer by using the application layer.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer vision technology, and in particular to a visual learning method, system, device, and storage medium. Background Technology

[0002] In recent years, with the accumulation of massive amounts of data, the improvement of computing power, and algorithmic innovation, artificial intelligence (AI) computer vision technologies have flourished. When using deep learning frameworks for model training, compatibility and adaptation between algorithms and frameworks, and between algorithms and datasets, are crucial. In related technologies, computer vision learning frameworks employ a variety of application programming interfaces (APIs), with inconsistent data formats and varying interface writing methods. Because the APIs of visual learning frameworks are written differently in different modes, training network models using different visual learning frameworks requires relearning, resulting in severe fragmentation and hindering the development of the AI ​​industry ecosystem. Summary of the Invention

[0003] In view of the above, embodiments of this application provide at least one visual learning method, system, device, and storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] On one hand, embodiments of this application provide a visual learning method, the method comprising:

[0006] The system resource layer provides the underlying data for visual learning;

[0007] An interface adaptation layer is used to call the underlying data from the system resource layer, and to provide interface services to the application layer based on the underlying data;

[0008] The visual learning is implemented using the application layer based on the interface services provided by the interface adaptation layer. On the other hand, embodiments of this application provide a visual learning system, which includes:

[0009] The system resource layer is used to provide the underlying data corresponding to visual learning;

[0010] The interface adaptation layer is used to call the underlying data from the system resource layer and provide interface services to the application layer based on the underlying data;

[0011] The application layer is used to implement the visual learning based on the interface services provided by the interface adaptation layer.

[0012] In another aspect, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0013] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0014] In another aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps for implementing the above-described method.

[0015] In another aspect, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0016] In this embodiment, firstly, the underlying data corresponding to visual learning is provided through the system resource layer, thus providing sufficient underlying data for the implementation process of visual learning. Then, the underlying data is called from the system resource layer through the interface adaptation layer, and interface services are provided to the application layer based on the underlying data. In this way, the interface adaptation layer is placed between the system resource layer and the application layer, which can not only call the required underlying data from the system resource layer, but also provide interface services to the application layer, which is the upper layer of the system. This allows the application layer to realize visual learning by calling the corresponding data from the interface services provided by the interface adaptation layer. In this way, the application layer can read different data through a unified interface adaptation layer, realizing the unified and standardized use of application interfaces. This allows the use of different deep learning frameworks as backends and the ability to switch between different deep learning frameworks, thereby improving the efficiency of visual learning.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 A schematic diagram illustrating the implementation process of a visual learning method provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram illustrating another implementation process of a visual learning method provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram illustrating another implementation process of a visual learning method provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram illustrating the implementation framework of the visual learning method provided in the embodiments of this application;

[0023] Figure 5 This is a schematic diagram of the composition structure of a visual learning system provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0028] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0029] 1) Computer vision refers to machine vision that uses cameras and computers to identify, track, and measure targets instead of human eyes, and further processes the images to make them more suitable for human observation or transmission to instruments for detection. In the embodiments of this application, computer vision refers to a functional unit with the ability to acquire, process, and interpret visual data.

[0030] 2) Machine learning, as an artificial neural network, can independently construct (train) basic rules based on example data during the learning process. Especially in the field of machine vision, neural networks are usually trained using supervised learning methods, that is, trained using example data and predefined results of the example data.

[0031] 3) Model training is the process of using training data and machine learning algorithms to establish or improve the parameters of a machine learning model.

[0032] 4) Training data is a subset of input data samples used to train machine learning models.

[0033] 5) Reasoning: Arguing and drawing conclusions from given premises. Reasoning refers to both the process and the result. In the field of artificial intelligence, a premise is a fact, a rule, a model, a feature, or raw data.

[0034] 6) Deep neural network learning is a method of creating rich hierarchical representations by training a neural network with many hidden layers.

[0035] This application provides a visual learning method that can be executed by a processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1 This is a schematic diagram illustrating the implementation flow of a visual learning method provided in an embodiment of this application. The method is applied to a visual learning system, which includes a system resource layer, an application layer, and an interface adaptation layer placed between the system resource layer and the application layer, such as... Figure 1 As shown, the method includes the following steps S101 to S103:

[0036] Step S101: The system resource layer provides the underlying data corresponding to visual learning.

[0037] In some embodiments, the system resource layer may be the underlying layer of the visual learning system, including an interface for communicating with the interface adaptation layer. The underlying data in the system resource layer includes both hardware data and software data, and can provide the storage space, image data, computation, and inference functions required by the visual learning system.

[0038] In some possible implementations, the system resource layer provides the machine learning framework (e.g., a deep learning framework) for implementing visual learning, the runtime environment for this framework to perform machine learning, the hardware device for implementing visual learning, and the computational scheduling of the hardware device. This computational scheduling can refer to the scheduling of threads, processes, or data streams within the device; for example, threads, processes, or data streams are scheduled onto hardware resources such as processors, network links, or expansion cards.

[0039] Step S102: The interface adaptation layer calls the underlying data from the system resource layer and provides interface services to the application layer based on the underlying data.

[0040] In some embodiments, the interface adaptation layer communicates with the system resource layer through software interfaces, playing a crucial bridging role in the visual learning framework. This interface adaptation layer retrieves underlying data from the system resource layer and provides interface services to the upper-layer system, i.e., the application layer. Specifically, it processes the underlying data through various interfaces within the interface adaptation layer, enabling the application layer to access the results of these processing methods. The interface services provided by the interface adaptation layer to the application layer include the services corresponding to each type of interface within the layer.

[0041] Step S103: The visual learning is implemented by the application layer based on the interface service provided by the interface adaptation layer.

[0042] In some embodiments, the application layer is an upper-level system within the visual learning framework, capable of performing processes such as training and inference of machine learning models. During model training, the application layer calls various interfaces defined in the interface adaptation layer, i.e., it calls the interface services provided by various interfaces in the interface adaptation layer, thereby enabling the training of the machine learning model and achieving visual learning.

[0043] In some embodiments, the application layer calls multiple interfaces in the interface adaptation layer during the model training phase to complete model training. That is, the application layer trains the machine learning model corresponding to the visual learning by calling multiple interfaces in the interface adaptation layer to achieve the visual learning. In this way, the training of the machine learning model corresponding to visual learning by the application layer can be achieved by calling multiple interfaces in the interface adaptation layer. Therefore, the later maintenance of this application layer can utilize different deep learning frameworks, meaning different deep learning frameworks can be used as backends.

[0044] In this embodiment, firstly, the underlying data corresponding to visual learning is provided through the system resource layer, thus providing sufficient underlying data for the implementation process of visual learning. Then, the underlying data is called from the system resource layer through the interface adaptation layer, and interface services are provided to the application layer based on the underlying data. In this way, the interface adaptation layer is placed between the system resource layer and the application layer, which can not only call the required underlying data from the system resource layer, but also provide interface services to the application layer, which is the upper layer of the system. This allows the application layer to realize visual learning by calling the corresponding data from the interface services provided by the interface adaptation layer. In this way, the application layer can read different data through a unified interface adaptation layer, realizing the unified and standardized use of application interfaces. This allows the use of different deep learning frameworks as backends and the ability to switch between different deep learning frameworks, thereby improving the efficiency of visual learning.

[0045] In some embodiments, multiple types of interfaces are defined in the interface adaptation layer to realize the calling relationship between the interface adaptation layer, the system resource layer, and the application layer. That is, the above step S102 can be implemented through the following process:

[0046] The system uses the various interfaces to call the underlying data from the system resource layer, and provides interface services to the application layer based on the underlying data through the various interfaces.

[0047] Here, the various interfaces within these multiple types have different functions. Each type of interface calls underlying data from the system resource layer, processes the underlying data within its own interface, and then provides interface services to the application layer through that interface. In this way, by defining multiple types of interfaces in the interface adaptation layer, these multiple types are used to call underlying data from the system resource layer and provide interface services to the application layer. This provides a unified interface for the application layer, enabling different deep learning frameworks to communicate with these multiple types of interfaces.

[0048] In some embodiments, the multiple types of interfaces in the interface adaptation layer include at least one of the following: model definition interface, algorithm training interface, and application inference interface. Thus, step S102 above can be achieved through... Figure 2 The steps shown are to be implemented as follows:

[0049] Step S201: Use the model definition class interface to provide the model structure corresponding to the underlying data.

[0050] In some embodiments, the model definition class interface is a required interface among multiple interface classes. The model definition class interface is used to define the data structure of the underlying data and to define a model structure that conforms to this data structure. The data structure of the underlying data includes the data size and data type (e.g., floating-point, integer), and supports direct computation of the data structure, such as initializing the underlying data, setting indexes for the underlying data, or slicing the underlying data. The model structure corresponding to the underlying data can be a functional module used to define the data structure conforming to the low-level data, supporting user-defined algorithm models.

[0051] In some possible implementations, by setting corresponding functions for the model definition interface, the data structure and model structure are defined through the model definition class interface. That is, the above step S201 can be implemented through the following steps S211 and S212 (not shown in the figure):

[0052] Step S211: Define the data structure of the underlying data using the model definition class interface.

[0053] Here, the model definition class interface supports defining the data structure and model structure of the underlying data. In other words, this model definition class interface allows users to define the data structure and model structure for computer vision. This data structure can be defined by the user through the model definition class interface. For example, during the development of a visual learning framework, developers can define a data interface through the model definition class interface, enabling the data in the framework to be transmitted in the form of this data structure.

[0054] Step S212: Determine the model structure based on the data structure.

[0055] Here, after defining this data structure, a model structure is set where both input and output are based on this data structure. This model structure allows for the definition of functional modules that implement visual learning, thus supporting users in defining algorithm models through this model definition class interface. In this way, by providing a model definition class interface in the interface adaptation layer, the basic data structures for computer vision and their corresponding model structures can be defined, thereby providing a unified data structure and model structure for the application layer.

[0056] Step S202: The algorithm training interface is used to train the model structure based on the underlying data, and the service corresponding to the algorithm training interface is provided to the application layer based on the trained model structure.

[0057] In some embodiments, by defining an algorithm training class interface in the interface adaptation layer, the model structure can be trained on the underlying data through the algorithm training class interface, thereby providing the application layer with the trained model structure so that the application layer can implement inference functions by calling the trained model structure.

[0058] In some embodiments, the algorithm training interface includes at least: a first data processing interface, an optimizer interface, a quantization training interface, a mixed precision interface, and a distributed interface. Training of the model structure is achieved through these interfaces; that is, step S202 can be implemented through the following steps S221 to S225 (not shown in the figures):

[0059] Step S221: Use the first data processing interface to convert the format of the underlying data into a format that matches the model structure to obtain the converted data.

[0060] Here, a first data processing interface is defined in the algorithm training class interface. This first data processing interface is used to convert the data format of the underlying data called from the system resource layer so that the converted data conforms to the data structure defined in the model definition class interface. For example, the underlying data is converted into tensors representing sample data, thereby enabling the loading of sample data for training the model structure from the system resource layer.

[0061] Step S222: The optimizer interface is used to update the model parameters of the model structure based on the transformed data, and the updated model structure is provided to the application layer.

[0062] Here, the optimizer interface uses the transformed data as sample data to adjust the model parameters of the model structure, such as adjusting the weights of the model structure, so as to obtain an updated model structure, so as to provide an updated model structure for the application layer.

[0063] Step S223: Use a quantization training interface to sense the loss of the model structure during the training process of the model structure.

[0064] Here, the transformed data is used as sample data to train the model structure defined by the model definition class interface, and the loss caused by model quantization is perceived through the quantization training interface.

[0065] Step S224: Using the mixed precision interface, perform mixed precision training on the updated model structure based on the loss, and provide the mixed precision trained model structure to the application layer.

[0066] Here, the mixed-precision interface can support mixed-precision training on the updated model structure to obtain the mixed-precision trained model structure, so that the application layer can call the mixed-precision trained model structure from the interface.

[0067] Step S225: The distributed interface is used to transmit the data of the model structure during the training process between multiple processes, and the data is transmitted to the application layer based on the multiple processes.

[0068] Here, the distributed interface transmits the data generated during the training process of the model structure in a parallel manner, and can also transmit data to the application layer through multiple parallel processes in the distributed interface.

[0069] Steps S221 to S225 above define interfaces such as the first data processing interface, optimizer interface, quantization training interface, mixed precision interface, and distributed interface in the algorithm training class interface, so that the interface adaptation layer can provide the application layer with the trained model structure and the results output by each interface during the training process.

[0070] In other embodiments, a visualization interface, a distillation interface, and a graph-to-calculation fusion interface can also be defined in the algorithm training interface. Thus, step S202 can be implemented through the following steps S226 to S228 (not shown in the figures):

[0071] Step S226: The underlying data is converted into a format that matches the model structure using the visualization interface, and the converted data is visualized and provided to the application layer.

[0072] Here, in the algorithm training interface, the input to the visualization interface can be the transformed data output from the first data processing interface, or it can be low-level data called from the system resource layer. The visualization interface visualizes the data generated during the training process.

[0073] Step S227: Using the distillation interface, the teacher network guides the student network to train the model structure, and provides the trained model structure to the application layer.

[0074] Here, the distillation interface supports training the model structure by having the teacher network guide the student network. It can be used in conjunction with the optimizer interface or mixed precision interface to quickly implement the training process of the model structure.

[0075] Step S228: The computational graph logic corresponding to the model structure is adjusted using the graph-computation fusion interface, and the model structure is adjusted synchronously based on the adjusted computational graph logic to provide the adjusted model structure for the application layer.

[0076] Here, the graph-computation fusion interface supports adjustments to the computational graph logic of the model structure, allowing the application layer to adjust this logic by calling the interface. Thus, by defining visualization, distillation, and graph-computation fusion interfaces within the algorithm training interface, the visualization interface can be used to visualize the data generated during training; the distillation interface can be used to distill the model structure, resulting in a smaller and higher-performing model; and the graph-computation fusion interface allows for timely adjustments to the computational graph logic, thereby optimizing the algorithm training interface.

[0077] Step S203: The underlying data is processed using the application reasoning interface, and the application layer is provided with the corresponding service of the application reasoning interface based on the processed underlying data.

[0078] In some embodiments, application inference interfaces can call underlying data from the system resource layer that needs to be inferred, and process that underlying data, such as performing image preprocessing. Application inference interfaces not only support the preprocessing of underlying data, but also support user plugin development and the use of various interfaces, thereby providing the application layer with processed underlying data. The processing of the underlying data by the application inference interfaces includes: visual task orchestration of the underlying data, preprocessing operations on the underlying data, and filtering of duplicate targets in the underlying data.

[0079] In some possible implementations, the application inference interface includes: a process orchestration interface, a block detection interface, a plug-in interface, and a second data processing interface. The image data is applied at the interface adaptation layer through these multiple application inference interfaces. That is, step S203 can be implemented through the following steps S231 to S234 (not shown in the figure):

[0080] Step S231: Use the second data processing interface to read the image data in the underlying data and preprocess the image data to obtain preprocessed image data, and provide the preprocessed image data to the application layer.

[0081] Here, the second data processing interface reads image data from the underlying data and performs preprocessing operations on the image data to obtain preprocessed image data. For example, it may adjust the image data sharpness or perform noise reduction on the image data to obtain preprocessed image data. The second data processing interface can provide this interface to the application layer, meaning that the application layer can call the preprocessed image data from the second data processing interface during model training.

[0082] Step S232: The preprocessed image data is arranged for visual tasks using the process orchestration interface to provide the orchestrated image data to the application layer.

[0083] Here, the preprocessed image data is input into the workflow orchestration interface, or the workflow orchestration interface obtains the preprocessed image data from the second data processing interface. The workflow orchestration interface performs visual task orchestration on the preprocessed image data, thereby providing orchestrated image data to the application layer, meaning the application layer can call the orchestrated image data from the workflow orchestration interface.

[0084] In some possible implementations, the workflow orchestration interface performs visual task orchestration on the preprocessed image data, including at least one of the following:

[0085] Method 1: Obtain the preprocessed image data; for example, call the preprocessed image data from the second data processing interface.

[0086] Method 2: Decode the preprocessed image data; for example, decode the preprocessed image data according to the decoding method required by the model structure defined in the model definition class interface, so that the application layer can call the decoded image data.

[0087] Method 3: Scale the preprocessed image data; for example, scale the preprocessed image data according to the standard size of the image data specified in the data structure defined in the model definition class interface, so that the scaled image data conforms to the data structure defined in the model definition class interface.

[0088] Method 4: Perform target detection on the preprocessed image data; for example, perform target detection on the preprocessed image data by calling the trained model structure in the algorithm training class interface to detect the target in the preprocessed image data; for example, if the preprocessed image data is a traffic scene, then the target can be a vehicle in the preprocessed image data.

[0089] Method 5: Crop the preprocessed image data; for example, crop the preprocessed image data according to the standard size of the image data specified in the data structure defined in the model definition class interface, or crop the region of interest in the preprocessed image data, etc.

[0090] Method 6: Perform image classification on the preprocessed image data; for example, perform image classification on the preprocessed image data by calling the trained model structure in the algorithm training class interface to determine the type to which the preprocessed image data belongs.

[0091] Method 7: Serialize the preprocessed image data. This converts the preprocessed image into a format that can be stored or transmitted.

[0092] Thus, visual task orchestration of preprocessed image data is achieved through methods one through seven, thereby enabling the provision of orchestrated image data to the application layer through a unified workflow orchestration interface.

[0093] Step S233: The block detection interface is used to filter overlapping targets in the preprocessed image data to provide filtered image data for the application layer.

[0094] In some embodiments, the block detection interface supports user-defined block counts, meaning users can define the number of blocks in the image data through the block detection interface. First, the image data is divided into blocks based on the defined block count using the block detection interface; then, the block detection interface is used to filter overlapping targets in the divided image data to obtain filtered image data.

[0095] Step S234: The preprocessed image data is processed using the plugin set in the plugin interface to obtain processed image data, and the processed image data is provided to the application layer.

[0096] In some embodiments, the plugin interface supports plugin development, registration, and compilation, and plugin development, registration, and compilation are performed through the plugin interface during the visual processing of the image data.

[0097] In steps S231 to S234 above, image data is obtained from the underlying data during the inference phase through multiple application inference class interfaces, and the image data is processed using each application inference class interface. This allows the application layer to provide the required image data through the application inference class interface, so that the application layer can obtain the processed image data by calling the application inference class interface in the interface adaptation layer.

[0098] In some embodiments, the system resource layer can provide both the hardware and software data required for visual learning, enabling the interface adaptation layer to process the underlying data through multiple interfaces, thereby providing interface services to the application layer. That is, step S101 described above can be achieved through... Figure 3 The steps shown are to be implemented as follows:

[0099] Step S301: Use the system resource layer to determine the hardware data and software data corresponding to the visual learning.

[0100] In some embodiments, hardware data includes the hardware device identifier corresponding to visual learning and the computational schedule of the hardware device, while software data includes image data, deep learning frameworks, and the operating environment of the deep learning frameworks.

[0101] In some possible implementations, step S301 can be achieved through the following steps:

[0102] The first step is to use the system resource layer to determine the hardware device identifier corresponding to the visual learning and the computation scheduling of the hardware device.

[0103] Here, the system resource layer analyzes the hardware device identifier corresponding to the visual learning process to determine the hardware device executing the visual learning. This hardware device identifier uniquely identifies the hardware device and can be a machine serial number. The hardware device's computational scheduling is used to characterize the process scheduling status within that hardware device.

[0104] The second step is to identify the hardware device identifier and the computing schedule as the hardware data.

[0105] In this way, by using the hardware device identifier and the computational scheduling of the hardware device as hardware data, sufficient hardware resources can be provided for the interface adaptation layer, thereby enabling the implementation of the functions of various interfaces in the interface adaptation layer based on these hardware resources.

[0106] Step S302: The hardware data and the software data are determined as the underlying data.

[0107] The above steps S301 and S302 provide hardware data and software data in the underlying data, thereby providing sufficient underlying data for the interface adaptation layer. This enables various interfaces in the interface adaptation layer to implement their functions by calling the underlying data from the system resource layer, facilitating the provision of various services to the application layer through a unified interface.

[0108] In some possible implementations, the machine learning framework, runtime environment, and image data during the visual learning process are analyzed at the system resource layer. That is, step S302 above can be achieved through the following steps:

[0109] The first step is to use the system resource layer to determine the machine learning framework corresponding to the visual learning, the operating environment of the machine learning framework, and the image data corresponding to the visual learning.

[0110] The second step is to identify the deep learning framework, the operating environment of the deep learning framework, and the image data as the software data.

[0111] The first and second steps described above determine the machine learning framework capable of visual learning and its operating environment through the system resource layer. At the same time, they determine the image data in the visual learning process and provide this data as software data to the interface adaptation layer, thereby providing sufficient software data for the visual learning framework.

[0112] In some embodiments, any visual learning task can be achieved by using a pre-built visual learning framework, which can be accomplished through the following steps:

[0113] The first step is to build a visual learning framework corresponding to the visual learning based on the system resource layer, the application layer, and the interface adaptation layer set between the system resource layer and the application layer.

[0114] The second step is to use the aforementioned visual learning framework to process the visual learning task corresponding to the visual learning.

[0115] By setting up the interface adaptation layer between the system resource layer and the application layer to build a visual learning framework, the application layer in the visual learning framework can call the unified interface in the interface adaptation layer, thereby achieving unified and standardized use of the interface.

[0116] The following describes the application of the visual learning method provided in the embodiments of this application in real-world scenarios, taking computer vision algorithms for deep learning as an example.

[0117] This application specifies the functional and technical requirements of the interfaces between artificial intelligence computer vision algorithms and deep learning frameworks, and between algorithms and datasets.

[0118] The embodiments of this application are applicable to the adaptation and invocation of application programming interfaces involved in the development and application of deep learning-based computer vision algorithms.

[0119] The Open Neural Network Exchange (ONNX) format, developed in related technologies, is a standard method for representing deep learning models, enabling model transfer between different frameworks (generally used in intermediate deployment phases). Hosting ONNX as open-source software on hosting platforms can encourage more frameworks to support it.

[0120] Different frameworks have limited compatibility and adaptability for algorithm training. With the development of the industry, major manufacturers have recognized the numerous problems caused by the strong coupling between frameworks and algorithms. In related technologies, computer vision learning frameworks have diverse APIs, inconsistent data formats, and different interface writing methods. The API writing methods differ under different modes. For example, an algorithm developed using one framework can only be upgraded and maintained using the same framework later. The relationship between the algorithm and the framework is tightly coupled, which is not conducive to secondary development.

[0121] Based on this, this application provides a visual learning method that adds an interface adaptation layer between the system resource layer and the algorithm application layer. Through the multiple types of interfaces in the added interface adaptation layer, the configured algorithm can use different deep learning frameworks as backends and can be switched; moreover, different data can be read in through a unified data interface and run on different hardware.

[0122] The visual learning method provided in the embodiments of this application is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the implementation framework of the visual learning method provided in the embodiments of this application. Figure 4 As can be seen, the framework includes: a system resource layer 401, an interface adaptation layer 402, and an algorithm application layer 403. The system resource layer 401 serves as the bottom layer, the interface adaptation layer 402 as the middle layer, and the algorithm application layer 403 as the top layer. Communication and calls between the system resource layer 401, the interface adaptation layer 402, and the algorithm application layer 403 are achieved through specific software interfaces. Here, the interface adaptation layer acts as a bridge, defining and implementing the functions and parameters involved in general interfaces (such as data structures and model structures), algorithm training interfaces (such as optimizers, distributed systems, quantization, mixed precision, and data processing), and algorithm inference interfaces (process orchestration, block detection). Specifically:

[0123] The system resource layer 401, comprising both hardware and software components, provides the necessary storage, computation, and inference functions for the computer vision system. The specific implementation of the system resource layer 401 varies among different manufacturers, and this embodiment will not further describe or specify it.

[0124] System resource layer 401 includes: a deep learning framework 411, a machine learning runtime environment 412, and hardware devices and their computation scheduling 413; wherein,

[0125] The Deep Learning Framework 411 provides a machine learning platform with a variety of training databases and community resources to help researchers advance advanced machine learning techniques and enable developers to easily build and deploy machine learning-powered applications.

[0126] Machine Learning Runtime Environment 412 provides an environment for running machine learning models.

[0127] Hardware devices and their computational scheduling 413, providing the hardware devices and computational scheduling required for the computer vision system.

[0128] The interface adaptation layer 402 provides service interfaces such as data interface, optimization interface, distributed interface, and model interface for the implementation process of the upper-layer system, ensuring that data, algorithms, and frameworks can be used for model training and model migration between different frameworks efficiently and flexibly.

[0129] In the interface adaptation layer 402, interfaces are divided into three main categories: model definition interface 41, algorithm training interface 42, and application inference interface 43. These interfaces are used to call resources from the system resource layer and also serve the upper-layer applications. Among them, the model definition interface is essential, mainly defining data structures and model structures, including: data structure 421 and model structure 422, where:

[0130] Data structure 421 refers to this data structure as a basic data structure (Tensor) in computer vision, which can support direct computation of basic data structures, such as initialization, indexing, slicing, etc.

[0131] Model structure 422 refers to a model structure whose input and output are both the basic data structure or a set of the basic data structure. For example, by defining a Module through the model structure, the framework should be able to support the definition of the algorithm model.

[0132] The interface for algorithm training 42 includes: optimizer interface 423, data processing interface 424, quantization training interface 425, distributed interface 426, mixed precision interface 427, and may also include visualization interface 428, distillation interface 429, and graph-computation fusion interface 430, among which...

[0133] The optimizer interface 423 is used to optimize the trained neural network model, that is, to optimize and update the parameters of the neural network model according to different optimization algorithms.

[0134] Data processing interface 424 supports converting data into the format required by the module, such as tensors.

[0135] The quantization training interface 425 provides the algorithm with the ability to perceive the information loss caused by model quantization during neural network training.

[0136] Distributed Interface 426 is used to complete the data transfer between multiple processes in a distributed training scenario with multiple machines and multiple GPUs.

[0137] The Mixed Precision Interface 427 provides unified support for mixed precision training of algorithms, which can reduce memory consumption and improve training speed when the graphics card supports it.

[0138] Visualization interface 428 is used to support the conversion of data into formats required by the module, such as tensors.

[0139] Distillation interface 429 is used to support the use of teacher networks to guide student networks during training, thereby improving the training accuracy of student models.

[0140] The Graph-Computation Fusion Interface 430 optimizes the overall network execution time by analyzing and optimizing the network computation graph logic, reducing the overhead of operator execution gaps, and improving the utilization of device computing resources.

[0141] The application inference interface 43 includes: a process orchestration interface 431, a block detection interface 432, a plug-in interface 433, and a data processing interface 434, among which,

[0142] The process orchestration interface 431 is used to compute visual task orchestration, such as image acquisition, image decoding, image scaling, object detection, image cropping, image classification, serialization, and other processes.

[0143] The block detection interface 432 is used to filter duplicate targets in overlapping areas after block division, and supports user-defined number of blocks.

[0144] Plugin Interface 433 is an interface for developing and using plugins that support the visual processing of computer vision systems, enabling users to develop, register, and compile plugins.

[0145] Data processing interface 434 is used for reading data from image files and performing preprocessing operations.

[0146] The algorithm application layer 403 can perform image recognition 451, image detection 452, image segmentation 453, and others by calling various interfaces in the interface adaptation layer.

[0147] The interface configuration method provided in this application can be used by computer vision algorithm developers, users, and evaluators. For example, if a computer vision user uses an algorithm from vendor A and finds that the algorithm no longer meets business needs, changing the algorithm would require considering the dependencies of related software. If the algorithm uses the aforementioned interface, there is less need to consider software and toolchain issues, making algorithm replacement and upgrades easier. Thus, in this application embodiment, by adding an interface adaptation layer between the system resource layer and the application inference layer, and by unifying and standardizing the use of algorithm application interfaces, the standardized use of computer vision algorithm interfaces is achieved, completing the decoupling of computer vision AI algorithms with learning frameworks and datasets. Furthermore, it allows for testing and evaluation of AI algorithms and models, solving the problem that testing platforms cannot be compatible with algorithms developed using different frameworks and cannot be evaluated on a unified platform. Moreover, it can guide the industry to implement algorithms according to interface requirements, thereby promoting interoperability between different training platforms.

[0148] In some embodiments, defining data structures through model definition class interfaces in the interface adaptation layer can be achieved through the following process:

[0149] Computer vision system interface requirements (requirements include mandatory and optional options; appropriate usage is recommended).

[0150] In some embodiments, the dataset can be in formats such as images, videos, or binary data. The dataset annotation file is in JSON format and contains annotation information for all samples in the dataset. If the annotation results include other auxiliary files, such as mask layer information, the relative paths of these auxiliary files are stored in the JSON file. Common data types are represented as follows:

[0151] Supports category labels, such as integers, where 0 represents the background and positive integers represent the foreground; supports bounding boxes, such as using the coordinates of the top-left and bottom-right vertices in the order (top-left x-coordinate, top-left y-coordinate, bottom-right x-coordinate, bottom-right y-coordinate); supports annotation files being parsed into lists or arrays after passing through the data reading interface, where each element is a dictionary or key-value pair container containing all relevant information for one of the samples, which can be accessed by the dataset using an index; supports loading specific dataset formats and supports custom datasets; supports annotation files supporting the data formats of mainstream open-source datasets.

[0152] In some embodiments, defining the model structure through the model definition class interface in the interface adaptation layer can be achieved through the following process:

[0153] a) Basic operators should be supported, including but not limited to "+", "-", "*", " / ", convolution operations, etc.; b) The meaning and values ​​of the parameters of the operator should be defined, and the computational logic for obtaining the output from the input during forward operation should be defined; c) Basic operators should support the construction of computational graphs through operator chaining and function nesting; d) The computational graph should support differentiation through backpropagation using the chain rule; e) The computational graph should support construction through conditional judgments or loops; f) User-defined operators should be compiled and allowed to be added to the computational graph; g) Serialization and deserialization of model parameters should be supported.

[0154] In this embodiment, the optimizer interface should support network model optimization, updating model parameters according to different optimization algorithms such as SGD, Adam, and Momomtum. Specifically, it should implement functions such as gradient calculation and backpropagation, parameter updating, optimization algorithm, and learning rate updating.

[0155] The optimizer interface should implement: a) support implementation as a class whose constructor parameters are the network model or a list of model parameters, as well as other required parameters, such as the learning rate; b) support implementation of the `step()` function for performing a single parameter optimization update. After this function is called, the model parameters should be updated based on the accumulated gradients.

[0156] In this embodiment, the mixed-precision training interface provides unified support for mixed-precision training of the algorithm, which can reduce memory consumption and improve training speed when supported by the graphics card. This interface should implement the following functions:

[0157] a) Supports precision conversion, converting model parameters to fp16, except for special layers (such as BN layers), while retaining a copy of the fp32 parameters; b) Supports input forward propagation, converting input data to fp16 for forward propagation and loss calculation; c) Supports loss amplification, amplifying the calculated loss with both fixed and dynamic amplification modes; d) Supports gradient calculation, calculating and backpropagating gradients in fp16 mode, then converting them to fp32 and scaling them back to the actual scale proportionally according to the amplification factor in c); e) Supports parameter update, updating parameters in the fp32 parameter copy based on the gradient calculated in d), and then assigning the updated parameters to the fp16 model.

[0158] In this embodiment, the framework can complete data transfer between multiple processes in a multi-machine, multi-GPU distributed training scenario through a distributed interface. This interface should have the following core functions:

[0159] The distributed interface should cover functions such as `bcast()`, `reduce()`, `scatter()`, `gather()`, `allreduce()`, `allgateher()`, and `sync()`. This set of interfaces should implement the following: support broadcasting data from the main process to each process; support reducing data from each process to the main process; support scattering a set of data from the main process to each process; support collecting data scattered across each process into the main process as a set; support reducing data from each process and then broadcasting it to each process; support collecting scattered data from each process into a set and then broadcasting it to each process; and support ensuring that all previously issued communication commands have been completed.

[0160] In this embodiment, the quantization training interface provides the algorithm with the ability to perceive the information loss caused by model quantization during neural network training. Specifically, during training, quantized weights are approximated using floating-point weights, allowing the quantization model to be simulated during forward propagation, calculating the floating-point error, and then backpropagating the error to update the weights. The quantization model helps accelerate model inference and reduce storage requirements. This interface should implement the following functions:

[0161] Supports input quantization, converting input from 32-bit floating-point type to 8-bit or custom-bit fixed-point type; supports quantization and dequantization of convolution and addition operators; supports pseudo-quantization nodes, which should include functions for quantizing and dequantizing floating-point weights; supports error backpropagation, calculating the error through b) during forward propagation, updating the floating-point weights, and then quantizing; supports quantized model output, and the quantized model can be converted into a fixed-point model for storage.

[0162] In this embodiment, a data interface supports the conversion of data into a format required by the module, such as a tensor. The data interface should have an outer interface and an inner interface. The framework should be able to prepare continuous training data for algorithm training in an iterative manner. This interface should have the following core functions:

[0163] Supports the implementation of an iterable data loader type, with each iteration returning a batch of data;

[0164] It supports sampling data from the dataset according to training requirements; it supports constructing dataset objects based on the dataset path and related parameters; it supports reading part or all of the dataset data from storage devices or services, such as annotation files and data samples; and it supports preprocessing operations on the data, such as image scaling, flipping, and color perturbation.

[0165] In this embodiment of the application, the training visualization interface provides the model structure, parameters, gradients, features, etc. during the algorithm training process. The interface should have the following features: support for visualization of model structure diagrams; support for visualization of feature maps; support for visualization of weight histograms; support for visualization of scalar changes; and support for visualization of convolutional kernels.

[0166] In this embodiment, the distillation interface supports the use of a teacher network to guide the student network during training, thereby improving the training accuracy of the student model: it should support the target distillation method; it should support the feature distillation method.

[0167] In this embodiment, graph-computation fusion optimizes the overall network execution time by analyzing and optimizing the existing network computation graph logic, reducing the overhead of operator execution gaps, and improving the utilization of device computing resources. It supports operations such as splitting, reorganizing, and merging the original computation logic; and it supports enabling graph-computation fusion by modifying the specified context parameter in the training script.

[0168] In this embodiment, the computer vision system supports process orchestration, and the interface meets the following requirements:

[0169] It should be able to combine key processes such as image acquisition, image decoding, image scaling, object detection, image cropping, image classification, and serialization;

[0170] It is advisable to support the plug-in approach for key processes, with configurable properties for each plug-in.

[0171] Users should be able to mount metadata.

[0172] It should support configuration file-based process orchestration management and have management components;

[0173] It should support specifying accelerators for particular processes;

[0174] It should support orchestration of multiple request and multiple output processes;

[0175] The following models should be supported for orchestration: YOLOv3, YOLOv3-tiny, ResNet50, Faster R-CNN, YOLOv4, SSD-VGG16, SSD MobileNet v1 FPN, CRNN, YOLOv5, Faster R-CNN-FPN / Cascade R-CNN-FPN, ResNet-18, DeepLabv3+, CTPN, DeepLabv3, BERT-Base (Uncased), DeepLabv3+, U-Net, Mask R-CNN, FaceNet, SSD MobileNet v1 FPN, OpenPose, Unet++, RetinaNet;

[0176] It should support single-input, single-output, multiple-input, and multiple-output orchestration.

[0177] In this embodiment, the computer vision system has a data processing interface, and the interface meets the following requirements:

[0178] It should support reading data from image files and moving it into a pre-configured cache;

[0179] It should support JPG / JPEG / BMP format image decoding, with a resolution range of (32*32, 8192*8192);

[0180] It should at least support JPG image encoding, with a resolution range of (32*32, 8192*8192);

[0181] Image scaling with specified target width and height should be supported, and image width and height should be scaled and aligned to the step size.

[0182] It should support specifying the expansion ratio in the four directions (top, bottom, left, and right) to expand the area of ​​the target bounding box for cropping.

[0183] It should support H264 / H265 video decoding, with a resolution range of (128*128, 4096*4096).

[0184] Width and height scaling alignment should be supported for step-based scaling.

[0185] It should support resolutions ranging from 128*128 to 1920*1920, H264 MP and H265 MP;

[0186] Image normalization, center cropping, affine transformation, and rotation should be supported;

[0187] It should support data transfer between key processes and preferably support the multiple distribution of a single input;

[0188] It should support data transfer between processor memory and main memory;

[0189] It should support frame skipping processing of video data;

[0190] Serialization should be supported.

[0191] In this embodiment, the computer vision system supports the development and use of visual processing plugins, and meets the following requirements: it should support user-developed plugins, registration, and compilation; it should support plugin initialization, deinitialization, execution, attribute registration, and acquisition interfaces; it should support interfaces for defining variable and immutable ports for plugin input and output; and it should support interfaces for defining and throwing business logic exceptions.

[0192] It should support streaming plugin interfaces to achieve the following functions: sending data of a specified category or channel to different ports; sequentially outputting input data from multiple ports through a single port; supporting multiple instantiation interfaces for plugins of the same type; supporting plugin caching mechanisms and interfaces to enable the transfer of business data (such as decoded video and image data) between plugins; supporting plugin metadata (such as classification information and target information) description interfaces, and implementing the transfer through plugin caching; supporting single-input, single-output, multi-input, and multi-output plugin interfaces; supporting inference plugin interfaces to support target classification, detection, and tensor-based (input) inference; and supporting model post-processing plugin interfaces to interface with models for target detection, classification, semantic segmentation, text generation, text box detection, pose detection, etc.

[0193] The application should support video analytics plugin interfaces to implement the following functions: multi-target path recording (including machine, non-machine, human, and face); face alignment (correcting detected face images); video quality diagnosis; and should support debugging plugin interfaces to enable data export (e.g., JSON format) and data loading and restoration. It should also support screen display plugin interfaces to enable drawing basic units on images, such as drawing frames, lines, circles, and writing text.

[0194] In this embodiment, the computer vision system supports a block detection interface and meets the following requirements:

[0195] It supports filtering duplicate targets in overlapping areas after segmentation; it supports user-defined parameters such as the number / size of segments and overlap, and automatically generates target boxes for image segments; it supports merging images of segmented inference results; and during multi-level inference, it supports filtering post-processing results based on the selection of maximum and minimum area, upper and lower area limits, and confidence thresholds.

[0196] A Module is a basic module in a neural network. Neural network modules build graphs based on this base class.

[0197] The Module provides Function 1: Forward Computation: a) Interface Name: forward; b) Interface Function Description: The module performs a forward computation and returns the computation result. If in training mode, a computation graph is constructed during the forward computation process to calculate the gradient of the module parameters.

[0198] Module provides function 2: Get trainable parameters of the module: a) Interface name: get_parameters; b) Interface function description: Returns the trainable parameters of the module.

[0199] The Module class provides function 3: Retrieving the module and its submodules: a) Interface name: get_modules; b) Interface function description: Optional. This interface returns an iterator that iterates through the module itself and its submodules, returning only one instance of a module.

[0200] Module provides function 4: Get module state: a) Interface name: get_state_dict; b) Interface function description: Returns the module state in key-value pairs, including the module parameters and buffer.

[0201] Module provides function 5: Load module state: a) Interface name: load_state_dict; b) Interface function description: Load the state of the module, including the module's parameters and buffers.

[0202] The Module provides Function 6: Module Backward Computation: a) Interface Name: backward(grad_input, grad_output); b) Interface Function Description: The module performs a backward computation and returns the computation result. If in training mode, the gradients of the module parameters are calculated during the backward computation process. This function is automatically generated by the computation graph and can also be registered later using register_backward_function.

[0203] Module provides function 7: Module forward computation: a) Interface name: register_backward_function; b) Interface function description: The module performs one backward computation and returns the module's computation result. If in training state, the gradient of the module parameters is calculated during the backward computation process.

[0204] Based on the foregoing embodiments, this application provides a visual learning system, which includes various units and modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0205] Figure 5 This is a schematic diagram of the composition structure of a visual learning system provided in an embodiment of this application, such as... Figure 5 As shown, the visual learning system 500 includes:

[0206] The system resource layer 501 is used to provide the underlying data corresponding to visual learning;

[0207] The interface adaptation layer 502 is used to call the underlying data from the system resource layer and provide interface services to the application layer based on the underlying data;

[0208] The application layer 503 is used to implement the visual learning based on the interface services provided by the interface adaptation layer.

[0209] In some embodiments, the interface adaptation layer includes multiple types of interfaces, and the interface adaptation layer 502 includes:

[0210] The various interfaces are used to call the underlying data from the system resource layer and provide interface services to the application layer based on the underlying data.

[0211] In some embodiments, the multiple interfaces include: model definition interface, algorithm training interface, and application inference interface.

[0212] In some embodiments, the interface adaptation layer 502 includes:

[0213] The model definition class interface is used to provide the model structure corresponding to the underlying data;

[0214] The algorithm training interface is used to train the model structure based on the underlying data to obtain the trained model structure, and to provide the application layer with the service corresponding to the algorithm training interface.

[0215] The application reasoning interface is used to process the underlying data and provide the application layer with the corresponding service based on the processed underlying data.

[0216] In some embodiments, the model definition class interface supports defining the data structure and model structure of the underlying data.

[0217] In some embodiments, the model definition class interface is further used to: define the data structure of the underlying data; and determine the model structure based on the data structure.

[0218] In some embodiments, the algorithm training interface includes: a first data processing interface, an optimizer interface, a quantization training interface, a mixed precision interface, and a distributed interface.

[0219] In some embodiments, the algorithm training interface includes:

[0220] The first data processing interface is used to convert the format of the underlying data into a format that matches the model structure, so as to obtain the converted data;

[0221] The optimizer interface is used to update the model parameters of the model structure based on the transformed data, and to provide the updated model structure to the application layer.

[0222] The quantization training interface is used to sense the loss of the model structure during the training process of the model structure.

[0223] The mixed-precision interface is used to perform mixed-precision training on the updated model structure based on the loss, and to provide the application layer with the mixed-precision trained model structure.

[0224] The distributed interface is used to transmit data of the model structure during the training process between multiple processes, and to transmit data to the application layer based on the multiple processes.

[0225] In some embodiments, the algorithm training interface further includes: a visualization interface, a distillation interface, and a graph-calculation fusion interface.

[0226] In some embodiments, the algorithm training interface includes:

[0227] The visualization interface is used to convert the format of the underlying data into a format that matches the model structure, visualize the converted data, and provide the application layer with visualized converted data.

[0228] The distillation interface is used to guide the student network to train the model structure through the teacher network and to provide the trained model structure to the application layer.

[0229] The graph-calculation fusion interface is used to adjust the computation graph logic corresponding to the model structure, and synchronously adjust the model structure based on the adjusted computation graph logic, so as to provide the adjusted model structure to the application layer.

[0230] In some embodiments, the application reasoning interface includes: a process orchestration interface, a block detection interface, a plug-in interface, and a second data processing interface.

[0231] In some embodiments, the application reasoning class interface includes:

[0232] The second data processing interface is used to read image data from the underlying data and preprocess the image data to obtain preprocessed image data, and to provide the preprocessed image data to the application layer;

[0233] The process orchestration interface is used to perform visual task orchestration on the preprocessed image data and provide orchestrated image data to the application layer.

[0234] The block detection interface is used to filter overlapping targets in the preprocessed image data and provide filtered image data to the application layer.

[0235] The plugin interface is used to process the preprocessed image data using the configured plugins to obtain processed image data, and to provide the processed image data to the application layer.

[0236] In some embodiments, the process orchestration interface is further configured to: acquire the preprocessed image data; perform image decoding on the preprocessed image data; scale the preprocessed image data; perform target detection on the preprocessed image data; crop the preprocessed image data; perform image classification on the preprocessed image data; and serialize the preprocessed image data.

[0237] In some embodiments, the system resource layer 501 is further configured to: determine the hardware data and software data corresponding to the visual learning; and determine the hardware data and the software data as the underlying data.

[0238] In some embodiments, the system resource layer 501 is further configured to: use the system resource layer to determine the hardware device identifier corresponding to the visual learning and the computing schedule of the hardware device; and determine the hardware device identifier and the computing schedule as the hardware data.

[0239] In some embodiments, the system resource layer 501 is further configured to: determine the machine learning framework corresponding to the visual learning, the operating environment of the machine learning framework, and the image data corresponding to the visual learning using the system resource layer; and determine the deep learning framework, the operating environment of the deep learning framework, and the image data as the software data.

[0240] In some embodiments, the application layer 503 is further configured to: train the machine learning model corresponding to the visual learning by calling multiple interfaces in the interface adaptation layer to achieve the visual learning.

[0241] In some embodiments, the system further includes:

[0242] A building module is used to build a visual learning framework corresponding to the visual learning based on the system resource layer, the application layer, and the interface adaptation layer set between the system resource layer and the application layer.

[0243] The processing module is used to process the visual learning task corresponding to the visual learning using the visual learning framework.

[0244] The descriptions of the system embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the system provided by this application can be used to execute the methods described in the method embodiments above. For technical details not disclosed in the system embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0245] It should be noted that, in the embodiments of this application, if the above-described visual learning method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0246] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0247] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0248] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0249] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0250] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0251] It should be noted that, Figure 6 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 6 As shown, the hardware entity of the computer device 600 includes: a processor 601, a communication interface 602, and a memory 603, wherein:

[0252] Processor 601 typically controls the overall operation of computer device 600.

[0253] Communication interface 602 enables computer devices to communicate with other terminals or servers via a network.

[0254] The memory 603 is configured to store instructions and applications executable by the processor 601, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) of the processor 601 and various modules in the computer device 600. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 601, the communication interface 602, and the memory 603 can be performed via bus 604.

[0255] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0256] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0257] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0258] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0259] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0260] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0261] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0262] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A visual learning method, characterized in that, The method includes: A system resource layer is used to provide the underlying data for visual learning; the underlying data includes hardware data and software data; the underlying data is used to provide the storage space, image data, computing and inference functions required by the visual learning system. An interface adaptation layer is used to call the underlying data from the system resource layer, and provides interface services to the application layer based on the underlying data; the interface adaptation layer includes multiple types of interfaces, including: model definition interfaces, algorithm training interfaces, and application inference interfaces; The visual learning is implemented by the application layer based on the interface services provided by the interface adaptation layer. The step of using an interface adaptation layer to call the underlying data from the system resource layer and providing interface services to the application layer based on the underlying data includes: The model structure corresponding to the underlying data is provided by the model definition class interface. The algorithm training interface is used to train the model structure based on the underlying data to obtain the trained model structure, and the service corresponding to the algorithm training interface is provided to the application layer. The underlying data is processed using the application inference class interface, and the service corresponding to the application inference class interface is provided to the application layer based on the processed underlying data.

2. The method according to claim 1, characterized in that, The model definition class interface supports defining the data structure and model structure of the underlying data.

3. The method according to claim 1, characterized in that, The provision of the model structure corresponding to the underlying data using the model definition class interface includes: The data structure of the underlying data is defined using the model definition class interface. Based on the data structure, the model structure is determined.

4. The method according to claim 1, characterized in that, The algorithm training interfaces include: a first data processing interface, an optimizer interface, a quantization training interface, a mixed precision interface, and a distributed interface.

5. The method according to claim 4, characterized in that, The algorithm training interface is used to train the model structure based on the underlying data to obtain the trained model structure, and the service corresponding to the algorithm training interface is provided to the application layer, including: The first data processing interface is used to convert the format of the underlying data into a format that matches the model structure, resulting in the converted data; The optimizer interface is used to update the model parameters of the model structure based on the transformed data, and the updated model structure is provided to the application layer. The quantization training interface is used to perceive the loss of the model structure during the training process of the model structure. The updated model structure is trained using the mixed-precision interface based on the loss, and the mixed-precision trained model structure is provided to the application layer. The distributed interface is used to transmit data of the model structure during training between multiple processes, and data is transmitted to the application layer based on the multiple processes.

6. The method according to claim 1, characterized in that, The algorithm training interfaces also include: visualization interface, distillation interface, and graph-calculation fusion interface.

7. The method according to claim 6, characterized in that, The algorithm training interface is used to train the model structure based on the underlying data to obtain the trained model structure, and the service corresponding to the algorithm training interface is provided to the application layer, including: The visualization interface is used to convert the format of the underlying data into a format that matches the model structure, and the converted data is visualized to provide the application layer with a visualized version of the converted data. The distillation interface is used to train the model structure through a teacher network and a student network, and to provide the trained model structure to the application layer. The computational graph logic corresponding to the model structure is adjusted using the graph fusion interface, and the model structure is adjusted synchronously based on the adjusted computational graph logic to provide the adjusted model structure for the application layer.

8. The method according to claim 1, characterized in that, The application reasoning interfaces include: process orchestration interface, block detection interface, plug-in interface, and second data processing interface.

9. The method according to claim 8, characterized in that, The step of processing the underlying data using the application inference class interface and providing the application layer with the service corresponding to the application inference class interface based on the processed underlying data includes: The image data in the underlying data is read using the second data processing interface and preprocessed to obtain preprocessed image data, which is then provided to the application layer. The preprocessed image data is arranged for visual tasks using the aforementioned process orchestration interface, providing the orchestrated image data to the application layer. The block detection interface is used to filter overlapping targets in the preprocessed image data, providing filtered image data to the application layer. The preprocessed image data is processed using the plugins set in the plugin interface to obtain processed image data, and the processed image data is provided to the application layer.

10. The method according to claim 9, characterized in that, The process of using the workflow orchestration interface to perform visual task orchestration on the preprocessed image data includes at least one of the following: Obtain the preprocessed image data; Image decoding is performed on the preprocessed image data; The preprocessed image data is scaled; Target detection is performed on the preprocessed image data; The preprocessed image data is cropped; The preprocessed image data is then classified. The preprocessed image data is then serialized.

11. The method according to any one of claims 1 to 10, characterized in that, The underlying data for visual learning provided by the system resource layer includes: The system resource layer is used to determine the hardware and software data corresponding to the visual learning. The hardware data and the software data are determined as the underlying data.

12. The method according to claim 11, characterized in that, The step of determining the hardware data corresponding to the visual learning using the system resource layer includes: The system resource layer is used to determine the hardware device identifier corresponding to the visual learning and the computational scheduling of the hardware device. The hardware device identifier and the computing schedule are identified as the hardware data.

13. The method according to claim 12, characterized in that, The system resource layer is used to determine the software data corresponding to the visual learning, including: The system resource layer is used to determine the machine learning framework corresponding to the visual learning, the operating environment of the machine learning framework, and the image data corresponding to the visual learning. The machine learning framework, the operating environment of the machine learning framework, and the image data are defined as the software data.

14. The method according to any one of claims 1 to 10, characterized in that, The implementation of visual learning using the application layer based on the interface services provided by the interface adaptation layer includes: The application layer uses multiple interfaces in the interface adaptation layer to train the machine learning model corresponding to the visual learning, thereby realizing the visual learning.

15. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Based on the system resource layer, the application layer, and the interface adaptation layer set between the system resource layer and the application layer, a visual learning framework corresponding to the visual learning is built. The visual learning framework described above is used to process the visual learning task corresponding to the visual learning.

16. A visual learning system, characterized in that, The visual learning system includes a system resource layer, an application layer, and an interface adaptation layer located between the system resource layer and the application layer. The visual learning system includes: The system resource layer is used to provide the underlying data corresponding to visual learning; the underlying data includes hardware data and software data; the underlying data is used to provide the storage space, image data, computing and inference functions required by the visual learning system; The interface adaptation layer is used to call the underlying data from the system resource layer and provide interface services to the application layer based on the underlying data; the interface adaptation layer includes multiple types of interfaces, including: model definition interfaces, algorithm training interfaces, and application inference interfaces; The application layer is used to implement the visual learning based on the interface services provided by the interface adaptation layer. The model definition class interface is used to provide the model structure corresponding to the underlying data; The algorithm training interface is used to train the model structure based on the underlying data to obtain the trained model structure, and to provide the application layer with the service corresponding to the algorithm training interface. The application reasoning interface is used to process the underlying data and provide the application layer with the corresponding service based on the processed underlying data.

17. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 15.

Images