A deployment method, device, storage medium and electronic device for oral data processing model on FPGA platform

Abstract

The present invention discloses a deployment method, device, storage medium and electronic device for an oral data processing model on an FPGA platform. The method includes: obtaining model description information of the oral data processing model, determining a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents the data flow relationship between nodes in the oral data processing model; converting the model description information of the oral data processing model into hardware description information; converting a hardware building block into a processing unit of an FPGA platform; and determining the connection relationship between processing units corresponding to a plurality of hardware building blocks based on constraint information corresponding to the data flow matrix, thereby forming a hardware deployment diagram corresponding to the oral data processing model. Based on the hardware deployment diagram, a hardware structure corresponding to the oral data processing model is constructed on the FPGA platform, so as to realize the processing function of the business type corresponding to the oral data processing model on the FPGA platform.

Description

Technical Field

[0001] The present invention relates to the field of neural network hardware acceleration technology, and in particular to a method, device, storage medium and electronic device for deploying an oral data processing model on an FPGA platform. Background Art

[0002] With the continuous development of deep learning models, deep learning models are widely used in various fields. Among them, deep learning models can be used to detect dental data to improve detection efficiency and accuracy.

[0003] As the complexity of deep learning models continues to increase, the amount of computation required has also increased significantly. General-purpose processors (such as CPUs) face challenges in speed and power consumption when handling load calculations. Although graphics processing units (GPUs) can provide high parallel computing capabilities, they are insufficient in terms of power consumption and hardware cost. Summary of the invention

[0004] The present invention provides a method, device, storage medium and electronic device for deploying an oral data processing model on an FPGA platform, which realizes the deployment of a hardware structure corresponding to the oral data processing model on the FPGA platform, deploys the oral data processing model on the FPGA platform, accelerates the execution efficiency of tooth detection, and reduces power consumption during the detection process.

[0005] According to one aspect of the present invention, a method for deploying an oral data processing model on an FPGA platform is provided, comprising:

[0006] Acquire model description information of the oral data processing model, and determine a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents a data flow relationship between nodes in the oral data processing model;

[0007] Converting the model description information of the oral data processing model into hardware description information; the hardware description information includes a plurality of hardware building blocks;

[0008] Converting the hardware building blocks into processing units of an FPGA platform;

[0009] Based on the constraint information corresponding to the data flow matrix, the connection relationship between the processing units corresponding to the multiple hardware building blocks is determined to form a hardware deployment diagram corresponding to the oral data processing model. The hardware deployment diagram is used to indicate the hardware structure corresponding to the oral data processing model to be built on the FPGA platform.

[0010] According to another aspect of the present invention, a deployment device for an oral data processing model on an FPGA platform is provided, comprising:

[0011] A data flow matrix determination module, used to obtain model description information of an oral data processing model, and determine a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents a data flow relationship between nodes in the oral data processing model;

[0012] A hardware description information determination module, used to convert the model description information of the oral data processing model into hardware description information; the hardware description information includes a plurality of hardware building blocks;

[0013] A processing unit conversion unit, used to convert the hardware building block into a processing unit of an FPGA platform;

[0014] A hardware deployment diagram forming module is used to determine the connection relationship between the processing units corresponding to the multiple hardware building blocks based on the constraint information corresponding to the data flow matrix, and form a hardware deployment diagram corresponding to the oral data processing model. The hardware deployment diagram is used to indicate the hardware structure corresponding to the oral data processing model to be built on the FPGA platform.

[0015] According to another aspect of the present invention, a method for processing oral data is provided, the method further comprising:

[0016] Acquire oral data of the target object;

[0017] Based on the hardware structure corresponding to the oral data processing model deployed on the FPGA platform, the oral data is processed to obtain the processing result corresponding to the oral data, wherein the hardware structure corresponding to the oral data processing model is implemented based on the deployment method of the oral data processing model on the FPGA platform described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided an oral data processing device, comprising:

[0019] An oral data acquisition module, used to acquire oral data of a target object;

[0020] A data processing module is used to process the oral data based on the corresponding hardware structure of the oral data processing model deployed on the FPGA platform to obtain the processing results corresponding to the oral data, wherein the corresponding hardware structure of the oral data processing model is implemented based on the deployment method of the oral data processing model on the FPGA platform described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for deploying an oral data processing model on an FPGA platform described in any embodiment of the present invention, or the oral data processing method.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for deploying an oral data processing model on an FPGA platform described in any embodiment of the present invention, or the oral data processing method when executed.

[0026] The technical solution of the embodiment of the present invention obtains the model description information of the oral data processing model, converts the model description information of the oral data processing model into hardware description information, and converts the hardware building blocks in the hardware description information into processing units on the FPGA platform, providing a hardware basis for realizing the conversion of the oral data processing model to the hardware structure on the FPGA platform. A data flow matrix is ​​obtained through the model description information of the oral data processing model to characterize the data flow relationship of the oral data processing model, and a connection relationship between processing units is established through the constraint information corresponding to the data flow matrix to obtain a hardware deployment diagram of the oral data processing model on the FPGA platform. Based on the hardware deployment diagram, a hardware structure corresponding to the oral data processing model can be built on the FPGA platform to realize the processing function of the business type corresponding to the oral data processing model on the FPGA platform, and further in the process of processing oral data through the hardware structure on the FPGA platform, the execution efficiency of tooth detection is accelerated and the power consumption during the detection process is reduced.

[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flow chart of a method for deploying an oral data processing model on an FPGA platform provided by an embodiment of the present invention;

[0030] Figure 2 is a flow chart of an oral treatment method provided by an embodiment of the present invention;

[0031] Figure 3 It is a structural schematic diagram of a deployment device for an oral data processing model on an FPGA platform provided by an embodiment of the present invention;

[0032] Figure 4 is a schematic structural diagram of an oral data processing device provided by an embodiment of the present invention;

[0033] Figure 5 It is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Figure 1This is a flowchart of a method for deploying an oral data processing model on an FPGA platform provided by an embodiment of the present invention. This embodiment can be applied to deploying an oral data processing model on an FPGA platform, and executing the process of processing tooth data by the oral data processing model through the hardware structure deployed on the FPGA platform, so as to achieve the effect of reducing the efficiency and energy saving of the tooth detection process. This method can be executed by a deployment device for an oral data processing model on an FPGA platform. The deployment device for an oral data processing model on the FPGA platform can be implemented in the form of hardware and / or software. The deployment device for an oral data processing model on the FPGA platform can be configured in an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a computer or a server. Figure 1 As shown, the method includes:

[0037] S110, obtaining model description information of the oral data processing model, and determining a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents a data flow relationship between nodes in the oral data processing model.

[0038] In this embodiment, the oral data processing model is a deep learning model for processing the oral data into a set business type. Here, the model type of the oral data processing model is not limited, for example, it can be a convolutional neural network model and a Transformer model. The oral data can be data collected by an oral scanning device, for example, it can be oral point cloud data, or it can also be a medical image such as a CBCT (Cone beam CT, cone beam CT) image. The processing of the set business type of the oral data includes but is not limited to classification processing, detection processing, etc. of the oral data, wherein the classification processing may include caries classification, tooth position classification, etc., and the detection processing may include but is not limited to dental plaque detection, etc. In some embodiments, the oral data processing model may include but is not limited to a caries detection model and a dental plaque detection model. Among them, the caries detection model can classify the oral data to obtain the classification result of the oral data in the caries dimension. The dental plaque detection model can detect the region of interest of the oral data, and the region of interest is the dental plaque area, and obtain the detection result with the dental plaque mark.

[0039] The oral data processing model is a trained deep learning model, which is trained based on the sample data of the set business type. Taking the set business type of dental plaque detection as an example, the sample data set may include oral sample data and dental plaque label data corresponding to the oral sample data. Taking the set business type of caries recognition as an example, the sample data set may include oral sample data and caries label data corresponding to the oral sample data. There is no limitation on the set business type here, and accordingly, the corresponding oral data processing model can be obtained by collecting the sample data set corresponding to the set business type for training.

[0040] The trained oral data processing model is converted into model description information, which is used to describe the structure and model parameters of the oral data processing model. Optionally, the model description information can be description information in the form of a script. Optionally, the oral data processing model is converted through a pre-set code conversion module to obtain model description information corresponding to the oral data processing model. For example, a code template is pre-set in the code conversion module, and the type and model parameters of each layer in the oral data processing model are identified, and the type and model parameters of the layer are written into the code template to obtain model description information. Optionally, the oral data processing model is converted through a pre-set description model to obtain model description information corresponding to the oral data processing model. The description model can be a neural network model, which is trained based on a sample model and the model description information of the sample model. The model description information corresponding to the oral data processing model is obtained by inputting the oral data processing model into the description model. Optionally, the oral data processing model and the editing area are displayed on the display page, and the model description information input by the user is received through the editing area.

[0041] The data flow matrix corresponding to the oral data processing model is used to characterize the data flow in the oral data processing model, for example, it may include the data flow rate and data flow direction of each node along the connection line in the oral data processing model. The data flow matrix includes multiple elements, and each element in the data flow matrix represents the data flow rate and data flow direction of a node along a connection line. The data flow rate can be represented by the numerical value of the element, and the data flow direction can be represented by the positive or negative value of the element.

[0042] The data flow matrix can be obtained by converting the model description information of the oral data processing model. Optionally, it can be realized by a preset neural network model; specifically, the data flow matrix can be obtained by inputting the model description information of the oral data processing model into the preset neural network model.

[0043] Optionally, determining a data flow matrix corresponding to the oral data processing model based on the model description information includes: generating a directed acyclic graph corresponding to the oral data processing model based on the model description information; the directed acyclic graph represents the nodes and connection relationships in the oral data processing model; determining a data flow matrix based on the directed acyclic graph, each element in the data flow matrix represents the data flow rate and data flow direction of a node along a connecting line.

[0044] A directed acyclic graph can be understood as a directed graph without loops, which is used to characterize the model structure of the oral data processing model. The directed acyclic graph includes nodes and connecting lines between nodes, and each node can represent a layer structure in the oral data processing model, wherein the layer structure in the oral data processing model includes but is not limited to convolution layer, pooling layer, activation function layer and normalization layer. The connecting lines between nodes are used to characterize the connection relationship between any two layer structures.

[0045] Optionally, the model description information of the oral data processing model is parsed by a data signal processor to obtain a directed acyclic graph corresponding to the oral data processing model. Optionally, the model description information of the oral data processing model is input into the graph generation model through a pre-set graph generation model to obtain a directed acyclic graph corresponding to the oral data processing model. It can be understood that the directed acyclic graph is provided with associated instantiation parameters, wherein the instantiation parameters associated with the nodes corresponding to different layer structures in the directed acyclic graph may be different, for example, the convolution layer instantiation parameters include the convolution kernel size of each filter in the convolution layer, the convolution kernel movement step size and the number of filters constituting the convolution layer filter group; the nonlinear layer (such as the activation function layer) instantiation parameters include the number of nonlinear units and the type of nonlinear function of the nonlinear layer; the pooling layer instantiation parameters include the pooling size, the step size between consecutive pooling windows, the number of pooling units in the pooling layer and the type of pooling operation to be applied.

[0046] The elements in the data flow matrix are determined based on the nodes and connecting lines in the directed acyclic graph. Specifically, each row in the data flow matrix corresponds to a connecting line, and each column corresponds to a node. The element (a, n) in the data flow matrix represents the data flow rate from node a along connecting line n, and the positive or negative value of the data flow rate represents the direction of the data flow. If the element is positive, it represents the flow rate of data generated by the node, and if the element is positive, it represents the flow rate of data consumed by the node.

[0047] Optionally, the data flow rate and data flow direction of each element can be determined according to the oral data processing model during the data processing process. Optionally, in the case of obtaining a trained oral data processing model, the test sample is processed based on the oral data processing model to obtain the data flow rate and data flow direction of each node in the oral data processing model, and stored. For example, a data flow file corresponding to the oral data processing model can be obtained, and the data flow file includes the data flow rate and data flow direction of each node in the oral data processing model. In the case of constructing a data flow matrix, a framework of a data flow matrix is ​​constructed based on a directed acyclic graph, that is, an initial matrix of A×N is obtained, and the data flow rate and data flow direction of each node in the oral data processing model are read from the pre-stored data flow file, and added to the above initial matrix to obtain a data flow matrix.

[0048] By setting the data flow matrix, the data flow direction of each node in the oral data processing model can be determined, providing a basis for the deployment of the oral data processing model on FPGA from the data flow dimension.

[0049] It is understandable that the oral data processing model includes at least one model branch, and the model description information includes branch description information of each model branch. Exemplarily, in the process of extracting features from oral data, feature extraction of different dimensions of oral data can be achieved through multiple model branches, such as but not limited to texture data feature branches and structural feature extraction branches. The structure of the oral data processing model is not limited here, and the oral data processing model can be constructed according to the business type and processing requirements of the oral data processing model.

[0050] For each branch description information, a directed acyclic graph of the model branch can be formed. Accordingly, the directed acyclic graph corresponding to the oral data processing model includes the directed acyclic graphs of each of the model branches. For the directed acyclic graph of each model branch, a data flow matrix of the model branch is formed, and accordingly, a data flow matrix of at least one of the model branches is obtained.

[0051] On the basis of the above-mentioned embodiment, the data flow matrices of each model branch are spliced ​​to obtain the data flow matrix corresponding to the oral data processing model. Optionally, the data flow matrices of each model branch are spliced ​​according to the connection relationship between the model branches. For example, the data flow matrices corresponding to different model branches can be spliced ​​in the form of up and down arrangement or left and right arrangement to form a new data flow matrix. Among them, the blank elements are filled with preset values, which can be 0 or 1. Exemplarily, for model branches with parallel relationships, the data flow matrices corresponding to the model branches with parallel relationships are spliced ​​in the form of up and down arrangement. For model branches with upstream and downstream relationships, the data flow matrices corresponding to the model branches with upstream and downstream relationships are spliced ​​in the form of left and right arrangement.

[0052] In this embodiment, each model branch is processed separately to obtain the data flow matrix corresponding to each model branch, and the data flow matrix corresponding to each model branch is spliced ​​to obtain the data flow matrix corresponding to the oral data processing model. Processing each model branch separately can simplify the processing process, and each model branch can be processed in parallel to improve processing efficiency.

[0053] S120, converting the model description information of the oral data processing model into hardware description information; the hardware description information includes a plurality of hardware building blocks.

[0054] It is understandable that FPGA is a programmable logic device that can perform general functions, that is, it can be programmed to implement certain logic processing functions. FPGA emerged as a semi-custom circuit in the field of application-specific integrated circuits, which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices. FPGA has higher integration, stronger logic functions and greater flexibility.

[0055] In this embodiment, by converting the oral data processing model into a hardware structure formed by a hardware processing unit on the FPGA platform, the oral data processing model is deployed on the FPGA platform, thereby improving the processing efficiency of the oral data processing model and reducing power consumption.

[0056] Optionally, the FPGA platform may include a description information conversion module, which can realize the conversion of software description information to hardware description information. The model description information of the oral data processing model is used as software description information, and the hardware description information of the oral data processing model is obtained by inputting the model description information of the oral data processing model into the information conversion module.

[0057] Optionally, through a pre-set machine learning model, which is used to realize the conversion of software description information into hardware description information, the hardware description information of the oral data processing model is obtained by inputting the model description information of the oral data processing model into the information conversion module.

[0058] Optionally, a mapping relationship between the software description information and the hardware description information is preset, and the mapping relationship may be a mapping relationship between the setting field information in the software description information and the hardware building block. The model description information of the oral data processing model is used as the software description information. By parsing the setting field information in the model description information of the oral data processing model, the hardware building block corresponding to the setting field information is determined based on the above mapping relationship. The hardware building blocks corresponding to each setting field information in the model description information of the oral data processing model form the hardware description information of the oral data processing model.

[0059] It can be understood that a hardware building block set is pre-set, which includes multiple hardware building blocks, for example, including but not limited to sliding window blocks, fork units, convolution and pooling mapping, non-linear mapping and memory I / O units, etc.

[0060] The sliding window block parameters include the number of sliding window units, the window size, and the sliding step of the window along the height and width of the feature map. The sliding window unit takes a pixel stream as input and outputs a window stream with a specified step size along the height and width of the input feature map. The fork unit parameters include the number of input streams, the size of the fork operation, and the number of characters per input stream. The fork operation receives the input data stream and copies them before outputting. The convolution and pooling map parameters include the number of calculation units and the input window normalization result. Each unit in the convolution and pooling map performs an operation to reduce the size of the window to a single value. For the convolution map, the operation of each of all units is the dot product between the input window and the corresponding weight. The input and output rates depend on the data flow rate of the unit. For both pooling operations, in the case of average pooling, the dot product unit is used with the convolution kernel, and in the case of maximum pooling, a comparator is used to find the maximum value. The nonlinear mapping parameters include the number of nonlinear units and the type of nonlinear function applied. The memory I / O unit mapping parameters include the number of used memory interfaces, the memory I / O subsystem data bus width, the theoretical maximum bandwidth of the memory I / O subsystem, and the normalized efficiency of the memory I / O subsystem.

[0061] By converting the model description information of the oral data processing model into hardware description information, description information is provided for deploying the oral data processing model on the FPGA platform.

[0062] S130, converting the hardware building block into a processing unit of an FPGA platform.

[0063] It is understandable that hardware building blocks such as convolution and pooling mapping, nonlinear mapping, etc. are not actual hardware processing units on the FPGA platform. By converting the above hardware building blocks into processing units of the FPGA platform, the conversion of the oral data processing model to the hardware structure on the FPGA platform is realized.

[0064] Optionally, the processing unit of the FPGA platform includes but is not limited to a multiplier and an adder. In some embodiments, the FPGA model selected by the user is obtained, and different FPGA models may correspond to different hardware resources. Preset the hardware resource description information corresponding to each FPGA model. The resource description information may include a set of parameters describing the available on-chip resources and off-chip memory characteristics, for example, including an on-chip resource set R and an off-chip resource set Rsc, and the on-chip resource set R includes elements such as a digital signal processor, a lookup table, a trigger, and on-chip storage resources. The off-chip resource set Rsc includes elements such as off-chip memory bandwidth, off-chip memory capacity, and dynamic random access registers.

[0065] The hardware building blocks include basic building blocks and non-basic building blocks; wherein the basic building blocks can be understood as the actual hardware processing units corresponding to the FPGA platform, and there is no need to convert the processing units, for example, the fork unit can correspond to the registers on the FPGA platform, etc. The non-basic building blocks can be understood as the actual hardware processing units that do not belong to the FPGA platform, and they need to be converted to the processing units of the FPGA platform. The non-basic building blocks include one or more of the convolution mapping blocks, the pooling mapping blocks and the non-thread mapping blocks.

[0066] Accordingly, converting the hardware building block into a processing unit of the FPGA platform includes: converting the non-basic building block into a processing unit of the FPGA platform. Optionally, a correspondence between the non-basic building block and the processing unit is pre-set, and the non-basic building block is converted into the processing unit based on the above correspondence. For example, a convolution map can be converted into a combination of a multiplier and an adder.

[0067] It is understandable that the layer structure included in the oral data processing model may be repeated, for example, it may include multiple convolution layers, multiple pooling layers, etc. Further, multiple convolution layers may include convolution layers with the same convolution kernel, such as 3×3 convolution layers, etc. For a convolution map that needs to be converted into a combination of multipliers and adders, the corresponding multiple convolution maps, pooling maps, and nonlinear maps in the oral data processing model require a large number of processing units to be deployed on the FPGA platform.

[0068] In order to simplify the hardware structure corresponding to the oral treatment model on the FPGA platform, the hardware building blocks corresponding to the oral treatment model are layer-folded to achieve the reuse of processing units on the FPGA platform and simplify the hardware structure.

[0069] Among them, layer folding includes the first folding between layers and the second folding of dot products within layers. The first folding between layers can be understood as the execution process of fully unfolding all operations on each layer, fully parallelizing all convolution operations in the filter bank, pooling operations in the pooling layer, and nonlinear operations in the nonlinear layer. The second fold of dot products within the layer can be understood as the pipeline expansion implementation of dot product operations within the convolution and average pooling units. The fully parallel implementation of dot products with an input size of N requires parallel operation of N multipliers, and the construction of an adder reduction tree with log2N levels and N-1 adders, including an additional adder to combine the output of the dot product with other results. The start interval of the pipeline expansion implementation is 1 cycle, and the throughput is 1 dot product per cycle. At the other end of the spectrum, a dot product unit can be implemented as a single MACC unit. All multiplication and accumulation operations are scheduled using time multiplexing, with a start interval equal to the length of each input vector, that is, N cycles, and the data throughput is reduced to 1 / n dot products per cycle.

[0070] Optionally, converting the non-basic building blocks into processing units of the FPGA platform includes: identifying non-basic building blocks of the same type among the non-basic building blocks; converting the non-basic building blocks of the same type into a group of processing units; wherein the non-basic building blocks include at least one dot product; the at least one dot product corresponds to a tree structure formed by a first number of first processing units and a second number of second processing units.

[0071] In the hardware description information of the oral data processing model, non-basic building blocks are classified into types to determine different types of basic building blocks. Optionally, the non-basic building blocks may include attribute information, for example, the attribute information may include the convolution kernel size of the convolution map, the attribute information of the pooling map may include the filter size, and the attribute information of the non-linear map may include the non-linear function type. The attribute information of the non-basic building blocks can be used to assist in the classification of non-basic building blocks. On the basis of dividing the non-basic building blocks into convolution maps, pooling maps and non-linear maps, further division processing is performed based on the attribute information of each non-basic building block, for example, convolution maps with the same attribute information are used as non-basic building blocks of the same type, pooling maps with the same attribute information are used as non-basic building blocks of the same type, and non-linear maps with the same attribute information are used as non-basic building blocks of the same type.

[0072] The same type of non-basic building blocks are converted into a group of processing units to achieve reuse of the group of processing units. When the number of the same type of non-basic building blocks is at least two, multiple non-basic building blocks of the same type reuse a group of processing units to reduce the occupation of hardware resources on the FPGA platform.

[0073] For any non-basic building block, the non-basic building block may include at least one dot product, and the dot products executed in parallel may be implemented by reusing multipliers. In this embodiment, the number of processing units required can be reduced while ensuring the processing function by reusing processing units between non-basic building blocks of the same type and between different dot products within the same non-basic building block. Among them, the first processing unit may be a multiplier, and the second processing unit may be an adder. Accordingly, the number of multipliers and adders may be determined by the number of dot products in the non-basic building block and / or the dot product input size.

[0074] On the basis of the above-mentioned embodiment, the hardware description information and the data flow matrix are displayed; in response to the partitioning operation on the data flow matrix, a plurality of data flow sub-matrices are obtained, and a corresponding relationship between each of the data flow sub-matrices and at least one hardware building block in the hardware description information is established; and the hardware building block corresponding to each of the data flow sub-matrices is converted into a processing unit of the FPGA platform.

[0075] In this embodiment, the data flow matrix and the hardware description information are displayed on the display page. For example, the display page includes two display areas, the first display area displays the data flow matrix, and the second display area displays the hardware description information. The first display area is an editable area, and a partition operation can be performed on the data flow matrix in the first display area to obtain multiple data flow sub-matrices. Each data flow sub-matrix can correspond to one or more hardware building blocks. For example, in response to the relationship establishing a connection, a corresponding relationship is established between the data flow sub-matrix and at least one hardware building block.

[0076] The data flow matrix is ​​divided according to the principle of complete parallelism of the data flow matrix corresponding to the oral processing model, and is implemented along the depth division method. The on-chip memory is used for data reuse and intermediate results, and the communication with the off-chip memory is kept to a minimum and is limited to the input and output streams required by the data flow sub-matrix. Different data flow sub-matrices are mapped to different bit streams, and different data flow sub-matrices correspond to different hardware frameworks. Any data flow sub-matrix corresponds to different hardware frameworks and includes at least one hardware building block.

[0077] For each hardware building block corresponding to the data stream sub-matrix, the non-basic building blocks in the hardware construction corresponding to the data stream sub-matrix are converted into processing units of the FPGA platform. For example, non-basic building blocks of the same type are determined in the hardware construction corresponding to the data stream sub-matrix, and the non-basic building blocks of the same type are converted into a group of processing units.

[0078] S140. Based on the constraint information corresponding to the data flow matrix, determine the connection relationship between the processing units corresponding to the multiple hardware building blocks, and form a hardware deployment diagram corresponding to the oral data processing model. The hardware deployment diagram is used to indicate the hardware structure corresponding to the oral data processing model to be built on the FPGA platform.

[0079] On the basis of converting the hardware building blocks into processing units, the connection relationship between the processing units corresponding to the multiple hardware building blocks is determined, and the processing units corresponding to the multiple hardware building blocks are connected based on the above connection relationship to obtain a hardware deployment diagram corresponding to the oral data processing model. Among them, the connection relationship between the processing units corresponding to the multiple hardware building blocks can be determined based on the connection relationship or data flow between the multiple hardware building blocks. It can be understood that for the hardware building blocks including basic building blocks and non-basic building blocks, in the case where the hardware building blocks are non-basic building blocks, the non-basic building blocks correspond to the processing units, and in the case where the hardware building blocks are basic building blocks, the hardware building blocks correspond to the hardware processing units of the FPGA platform, and the hardware processing units can be such as registers, memories, triggers or processors.

[0080] In some embodiments, the connection relationship between the processing units corresponding to multiple hardware building blocks can be determined based on the constraint information corresponding to the data flow matrix, wherein the constraint information corresponding to the data flow matrix can be understood as the constraint information of the data flow dimension, for example, including the connection relationship and the data flow direction, and the data flow direction can characterize the connection relationship and flow direction of the data between each layer structure in the oral data processing model.

[0081] Optionally, based on the constraint information corresponding to the data flow matrix, the connection relationship between the processing units corresponding to the multiple hardware building blocks is determined, including: based on a preset search algorithm, taking the data flow matrix as constraint information and taking the shortest connection path as the goal, performing a connection search on the processing units corresponding to the basic building blocks and the non-basic building blocks to determine the connection relationship between the basic building blocks and the processing units.

[0082] In some embodiments, the data flow matrix may be used as constraint information; in some embodiments, the constraint information may be extracted from the data flow matrix, and the constraint information may be extracted by the magnitude and positive and negative values ​​of the elements in the data flow matrix. For example, the constraint information may be input into a constraint information extraction model to obtain the constraint information corresponding to the data flow matrix.

[0083] Through a preset search algorithm, the data flow matrix is ​​used as constraint information, and the shortest connection path is used as the goal to perform a connection search on the processing units corresponding to the basic building blocks and the non-basic building blocks, so as to obtain the connection relationship between the basic building blocks and the processing units, and form a hardware deployment diagram corresponding to the oral data processing model through the connection relationship between the basic building blocks and the processing units. The hardware deployment diagram may include basic building blocks and processing units, and the connection relationship between the basic building blocks and the processing units, and the basic building blocks and the processing units are hardware processing units on the FPGA platform, and the connection relationship between the hardware processing units on the FPGA platform is formed.

[0084] Optionally, the preset search algorithm may include but is not limited to a simulated annealing algorithm and an ant colony algorithm, etc., which are not limited here.

[0085] The hardware processing units on the FPGA platform can be connected through the hardware deployment diagram to obtain the hardware structure corresponding to the oral data processing model on the FPGA platform. The hardware structure can realize the processing function of the business type corresponding to the oral data processing model. For example, when the oral data processing model is a dental plaque detection model, the hardware structure corresponding to the oral data processing model on the FPGA platform can realize dental plaque detection.

[0086] The technical solution provided in this embodiment obtains the model description information of the oral data processing model, converts the model description information of the oral data processing model into hardware description information, and converts the hardware building blocks in the hardware description information into processing units on the FPGA platform, providing a hardware foundation for realizing the conversion of the oral data processing model to the hardware structure on the FPGA platform. A data flow matrix is ​​obtained through the model description information of the oral data processing model to characterize the data flow relationship of the oral data processing model, and a connection relationship between processing units is established through the constraint information corresponding to the data flow matrix to obtain a hardware deployment diagram of the oral data processing model on the FPGA platform. Based on the hardware deployment diagram, the hardware structure corresponding to the oral data processing model can be built on the FPGA platform to realize the processing function of the business type corresponding to the oral data processing model on the FPGA platform. Further, in the process of processing oral data through the hardware structure on the FPGA platform, the execution efficiency of tooth detection is accelerated and the power consumption during the detection process is reduced.

[0087] Figure 2 This is a flow chart of an oral treatment method provided by an embodiment of the present invention. Based on the above embodiment, the oral data processing model corresponding to the hardware structure is deployed on the FPGA platform, and the oral data of the target object is processed by the hardware structure deployed on the FPGA platform to obtain the processing result. Figure 2 , the method specifically comprises:

[0088] S210: Acquire oral data of the target object.

[0089] S220 , based on the hardware structure corresponding to the oral data processing model deployed on the FPGA platform, process the oral data to obtain a processing result corresponding to the oral data.

[0090] In some embodiments, the oral cavity data of the target object may be oral cavity point cloud data.

[0091] Optionally, based on obtaining the oral point cloud data of the target object, the oral point cloud data is preprocessed, wherein the preprocessing includes removing noise points in the oral point cloud data. It is understandable that noise points may be generated due to environmental factors (external light interference, saliva reflection), which may interfere with the processing of the oral data.

[0092] For example, the oral point cloud data may be filtered to remove noise points in the oral point cloud data. The filtering algorithm may include a radius filtering algorithm, which is used to filter out noise points, define a radius range for each data point in the oral point cloud data, determine the number of adjacent data points and the distribution density variation within the radius range, determine the distribution density variation value, and determine the number of adjacent data points. If the distribution density variation is greater than a first threshold and / or the number of data points is less than a second threshold, the point is determined to be a noise point and the noise point is removed.

[0093] In some embodiments, after obtaining the processing result corresponding to the oral data, the processing result corresponding to the oral data is further displayed. Taking the oral data being processed for dental plaque detection as an example, the processing result corresponding to the oral data may be a dental plaque detection result, which may be in the form of an image.

[0094] The FPGA platform is provided with an image display module, which uses FPGA to drive the VGA interface to display the processing results corresponding to the oral data. The image display module is used to implement one or more of display mode setting, pixel clock generation, image data storage and reading, synchronization signal generation, display optimization and debugging.

[0095] The display mode setting includes resolution selection and color depth determination. For example, the resolution is 640×480, the pixel clock frequency is about 25MHz, and the period and pulse width of the line synchronization signal and the field synchronization signal are determined. The color depth can be 16 bits (65536 colors) to load the color depth of the display image. The display mode setting page can be displayed, and the setting items of resolution and color depth are displayed on the display mode setting page. According to the user's setting operation of resolution and color depth, the target resolution and target color depth are determined.

[0096] Pixel clock generation includes clock module design and clock stability and accuracy. The clock module design uses the clock management unit inside the FPGA to multiply or divide the external input clock to obtain the appropriate pixel clock frequency, from a lower frequency external clock source (50MHz), multiply it to 25MHz through PLL for 640×480 resolution VGA display. Clock stability and accuracy When designing the clock module, ensure that the jitter of the clock signal is within an acceptable range and the frequency error is reduced as much as possible.

[0097] Image data storage and reading include storage resource utilization and data reading order. Storage resources use external memory (DDR SDRAM) to read image data and cache it in BRAM. When storing image data, the storage capacity and data format are determined according to the selected resolution and color depth. The data reading order reads the image data in the order of VGA display. In each pixel clock cycle, the RGB data of one pixel is read from the storage resource and output to the RGB signal channel of the VGA interface. The order of reading data is to start from the upper left corner of the image and read line by line from left to right and from top to bottom. For an image with a resolution of 640×480, the data of 640 pixels in the first line is read first, then the second line is read, and so on, until all 480 lines are read.

[0098] The synchronization signal generation includes the line synchronization signal (HSync) and the field synchronization signal (VSync). At the beginning of each line cycle, the line synchronization signal will generate a negative pulse to indicate that the display starts to display a new line of pixels. At a resolution of 640×480, the line cycle is about 31.77μs, and the line synchronization pulse width is about 3.81μs. The field synchronization signal refresh rate is set to 60Hz. At the beginning of each field cycle, the field synchronization signal will generate a negative pulse to indicate that the display starts to refresh a new frame of the image. At a resolution of 640×480, the field cycle is about 16.68ms, and the field synchronization pulse width is about 63.7μs.

[0099] Image display optimization and debugging include bandwidth optimization and debugging methods. Bandwidth optimization uses data caching and pre-fetching to reduce data transmission delays and bottlenecks, and reads the image data of the next line or lines into the cache in advance so that it can be quickly output when needed. The debugging method uses the FPGA development tool oscilloscope (Logic Analyzer) function to monitor the waveforms of RGB signals, synchronization signals, and pixel clock signals. By observing whether the waveforms meet the standards and design requirements of the VGA interface, it is determined whether the system is working properly. If the image display is incomplete or the color is wrong, the signal waveform is checked to locate whether the data is read correctly and whether the synchronization signal is accurate.

[0100] The technical solution of this embodiment realizes the processing of oral data through the hardware structure deployed on the FPGA platform, thereby accelerating the execution efficiency of tooth detection and reducing power consumption during the detection process.

[0101] Figure 3 FIG. 1 is a schematic diagram of a structure of a deployment device for an oral data processing model on an FPGA platform provided by an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0102] A data flow matrix determination module 310 is used to obtain model description information of an oral data processing model, and determine a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents a data flow relationship between nodes in the oral data processing model;

[0103] A hardware description information determination module 320 is used to convert the model description information of the oral data processing model into hardware description information; the hardware description information includes a plurality of hardware building blocks;

[0104] A processing unit conversion unit 330, configured to convert the hardware building block into a processing unit of an FPGA platform;

[0105] The hardware deployment diagram forming module 340 is used to determine the connection relationship between the processing units corresponding to the multiple hardware building blocks based on the constraint information corresponding to the data flow matrix, and form a hardware deployment diagram corresponding to the oral data processing model. The hardware deployment diagram is used to indicate the hardware structure corresponding to the oral data processing model to be built on the FPGA platform.

[0106] Based on the above embodiment, optionally, the data flow matrix determination module 310 is used to: generate a directed acyclic graph corresponding to the oral data processing model based on the model description information; the directed acyclic graph represents the nodes and connection relationships in the oral data processing model; determine the data flow matrix based on the directed acyclic graph, and each element in the data flow matrix represents the data flow rate and data flow direction of a node along a connecting line.

[0107] Optionally, the oral data processing model includes at least one model branch, the model description information includes branch description information of each model branch; the directed acyclic graph corresponding to the oral data processing model includes the directed acyclic graph of each model branch; the data flow matrix includes the data flow matrix of each model branch;

[0108] The data flow matrix determination module 310 is further used to: splice the data flow matrices of each of the model branches.

[0109] Based on the above embodiment, the hardware building block includes a basic building block and a non-basic building block; the non-basic building block includes one or more of a convolution mapping block, a pooling mapping block and a non-thread mapping block;

[0110] The processing unit conversion unit 330 is used to convert the non-basic building block into a processing unit of the FPGA platform.

[0111] Optionally, the processing unit conversion unit 330 is also used to: identify non-basic building blocks of the same type in the non-basic building blocks; convert the non-basic building blocks of the same type into a group of processing units; wherein the non-basic building blocks include at least one dot product; the at least one dot product corresponds to a tree structure formed by a first number of first processing units and a second number of second processing units.

[0112] Optionally, the processing unit conversion unit 330 is also used to: display the hardware description information and the data flow matrix; obtain multiple data flow sub-matrices in response to a partitioning operation on the data flow matrix, and establish a corresponding relationship between each of the data flow sub-matrices and at least one hardware building block in the hardware description information; and convert the hardware building block corresponding to each of the data flow sub-matrices into a processing unit of the FPGA platform.

[0113] Optionally, the hardware deployment diagram forming module 340 is used to: based on a preset search algorithm, use the data flow matrix as constraint information, take the shortest connection path as the goal, perform a connection search on the processing units corresponding to the basic building blocks and the non-basic building blocks, and determine the connection relationship between the basic building blocks and the processing units.

[0114] The deployment device of the oral data processing model on the FPGA platform provided in the embodiment of the present invention can execute the deployment method of the oral data processing model on the FPGA platform provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0115] Figure 4 Schematic diagram of the structure of an oral data processing device provided by an embodiment of the present invention. Figure 4 As shown, the device comprises:

[0116] An oral data acquisition module 410 is used to acquire oral data of a target object;

[0117] The data processing module 420 is used to process the oral data based on the hardware structure corresponding to the oral data processing model deployed on the FPGA platform to obtain the processing results corresponding to the oral data.

[0118] Optionally, the device further comprises a preprocessing module, which is used to preprocess the oral data of the target object, wherein the preprocessing is used to remove noise points in the oral data of the target object;

[0119] Optionally, the device further includes an image display module, and the image display module is used to display the processing results corresponding to the oral data.

[0120] The oral data processing device provided in the embodiment of the present invention can execute the oral data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Figure 5 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0122] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0124] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a deployment method of an oral data processing model on an FPGA platform, or an oral data processing method.

[0125] In some embodiments, the method for deploying an oral data processing model on an FPGA platform, or the method for deploying oral data processing can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, the method for deploying an oral data processing model on the FPGA platform described above, or one or more steps of the method for deploying oral data processing can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the method for deploying an oral data processing model on the FPGA platform, or the method for processing oral data, by any other appropriate means (for example, by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The computer program for implementing the method for deploying the oral data processing model on the FPGA platform of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a method for deploying an oral data processing model on an FPGA platform, the method comprising:

[0129] Acquire model description information of an oral data processing model, determine a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents the data flow relationship between nodes in the oral data processing model; convert the model description information of the oral data processing model into hardware description information; the hardware description information includes a plurality of hardware building blocks; convert the hardware building blocks into processing units of an FPGA platform; determine a connection relationship between processing units corresponding to the plurality of hardware building blocks based on constraint information corresponding to the data flow matrix, and form a hardware deployment diagram corresponding to the oral data processing model, wherein the hardware deployment diagram is used to indicate the construction of a hardware structure corresponding to the oral data processing model on the FPGA platform.

[0130] Alternatively, the computer instructions are used to cause the processor to execute an oral data processing method, which includes: an oral data acquisition module, used to acquire the oral data of the target object; a data processing module, used to process the oral data based on the hardware structure corresponding to the oral data processing model deployed on the FPGA platform, and obtain the processing results corresponding to the oral data.

[0131] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0134] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for deploying an oral data processing model on an FPGA platform, characterized in that: include: Acquire model description information of the oral data processing model, and determine a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents a data flow relationship between nodes in the oral data processing model; Converting the model description information of the oral data processing model into hardware description information; the hardware description information includes a plurality of hardware building blocks; Converting the hardware building blocks into processing units of an FPGA platform; Based on the constraint information corresponding to the data flow matrix, the connection relationship between the processing units corresponding to the multiple hardware building blocks is determined to form a hardware deployment diagram corresponding to the oral data processing model. The hardware deployment diagram is used to indicate the deployment of the hardware structure corresponding to the oral data processing model on the FPGA platform.

2. The method according to claim 1, characterized in that The step of determining a data flow matrix corresponding to the oral data processing model based on the model description information includes: Generating a directed acyclic graph corresponding to the oral data processing model based on the model description information; the directed acyclic graph represents the nodes and connection relationships in the oral data processing model; A data flow matrix is ​​determined based on the directed acyclic graph, wherein each element in the data flow matrix represents a data flow speed and a data flow direction of a node along a connection line.

3. The method according to claim 2, characterized in that The oral data processing model includes at least one model branch, and the model description information includes branch description information of each of the model branches; The directed acyclic graph corresponding to the oral data processing model includes the directed acyclic graphs of each of the model branches; The data flow matrix includes the data flow matrix of each model branch; The method further comprises: splicing the data flow matrices of each of the model branches.

4. The method according to claim 1, characterized in that: The hardware building blocks include basic building blocks and non-basic building blocks; the non-basic building blocks include one or more of a convolutional mapping block, a pooling mapping block, and a non-threaded mapping block; The processing unit for converting the hardware building block into an FPGA platform includes: The non-basic building blocks are converted into processing units of the FPGA platform.

5. The method according to claim 4, characterized in that The processing unit for converting the non-basic building block into an FPGA platform includes: identifying non-basic building blocks of the same type among the non-basic building blocks; The non-basic building blocks of the same type are converted into a group of processing units; wherein the non-basic building blocks include at least one dot product; and the at least one dot product corresponds to a tree structure formed by a first number of first processing units and a second number of second processing units.

6. The method according to claim 1 or 4, characterized in that: The method further comprises: Displaying the hardware description information and the data flow matrix; In response to the partitioning operation on the data flow matrix, a plurality of data flow sub-matrices are obtained, and a corresponding relationship between each of the data flow sub-matrices and at least one hardware building block in the hardware description information is established; The hardware building block corresponding to each of the data stream sub-matrices is converted into a processing unit of the FPGA platform.

7. The method according to claim 4, characterized in that The determining, based on the constraint information corresponding to the data flow matrix, the connection relationship between the processing units corresponding to the plurality of hardware building blocks comprises: Based on a preset search algorithm, with the data flow matrix as constraint information and the shortest connection path as the goal, a connection search is performed on the processing units corresponding to the basic building blocks and the non-basic building blocks to determine the connection relationship between the basic building blocks and the processing units.

8. A method for processing oral data, characterized in that: The method further comprises: Acquire oral data of the target object; Based on the hardware structure corresponding to the oral data processing model deployed on the FPGA platform, the oral data is processed to obtain the processing results corresponding to the oral data, wherein the hardware structure corresponding to the oral data processing model is implemented based on the deployment method of the oral data processing model on the FPGA platform described in any one of claims 1-7.

9. A deployment device for an oral data processing model on an FPGA platform, characterized in that: include: A data flow matrix determination module, used to obtain model description information of an oral data processing model, and determine a data flow matrix corresponding to the oral data processing model based on the model description information, wherein the data flow matrix represents a data flow relationship between nodes in the oral data processing model; A hardware description information determination module, used to convert the model description information of the oral data processing model into hardware description information; the hardware description information includes a plurality of hardware building blocks; A processing unit conversion unit, used to convert the hardware building block into a processing unit of an FPGA platform; A hardware deployment diagram forming module is used to determine the connection relationship between the processing units corresponding to the multiple hardware building blocks based on the constraint information corresponding to the data flow matrix, and form a hardware deployment diagram corresponding to the oral data processing model. The hardware deployment diagram is used to indicate the hardware structure corresponding to the oral data processing model to be built on the FPGA platform.

10. An oral data processing device, characterized in that: include: An oral data acquisition module, used to acquire oral data of a target object; A data processing module is used to process the oral data based on the corresponding hardware structure of the oral data processing model deployed on the FPGA platform to obtain the processing results corresponding to the oral data, wherein the corresponding hardware structure of the oral data processing model is implemented based on the deployment method of the oral data processing model on the FPGA platform described in any one of claims 1-7.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for deploying an oral data processing model on the FPGA platform described in any one of claims 1-7, or the oral data processing method described in claim 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for deploying an oral data processing model on an FPGA platform according to any one of claims 1 to 7, or the oral data processing method according to claim 8 when executed.

Images