Multi-neural network model calling method and device, industrial camera and storage medium

Abstract

The multi-neural network model calling method and device, industrial camera and storage medium provided in the present application relate to the field of neural network model calling technology. In the present application, firstly, in response to the multi-neural network model configuration operation performed, the multiple neural network models that need to be called are called and configured to form a target calling configuration file; secondly, in response to the neural network model calling operation performed, the target calling configuration file is loaded to complete the loading of multiple neural network models, and the target industrial camera is triggered to start to perform image acquisition operations to obtain image data to be processed; then, the image data to be processed is loaded into multiple neural network models, so that the multiple neural network models process the image data to be processed in turn, and return the image data processing results. Based on the above content, the problem of high complexity of the implementation logic of multi-neural network model calling in the prior art can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of neural network model calling, and in particular to a method and device for calling multiple neural network models, an industrial camera, and a storage medium. Background Art

[0002] When industrial cameras inspect items in actual business scenarios, they need to run corresponding algorithm applications. The application includes code to implement corresponding data pre-processing, model loading, model reasoning, and data post-processing, and finally obtain the results and display them on the web interface to achieve an intuitive viewing effect. Currently, there are six independent scenarios in industrial scenarios, including positioning, segmentation, positive samples, target detection, OCR recognition, and classification. However, for some case scenarios, the models need to be combined. For example, to detect defects in a part of the chip, it is necessary to first obtain the image data of the specified area through the positioning model, and then identify the defects in the area through the segmentation model and mark them, and finally obtain the result map after positioning + segmentation.

[0003] However, in the prior art, for different models, it is necessary to perform separate application configuration for model loading and model reasoning, and recompile, package, and deploy them before they can be used, which cannot achieve the effect of free combination of models. In other words, in the prior art, there is a problem of high complexity in the implementation logic of calling multiple neural network models. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a multi-neural network model calling method and device, an industrial camera and a storage medium, so as to improve the problem of high complexity of the implementation logic of multi-neural network model calling existing in the prior art.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] A multi-neural network model calling method is applied to an industrial camera, and the multi-neural network model calling method includes:

[0007] In response to the multi-neural network model configuration operation performed by the target user, a call configuration process is performed on the multiple neural network models that need to be called to form a corresponding target call configuration file, wherein the target call configuration file at least has a call sequence of the multiple neural network models that need to be called and a call path of each neural network model;

[0008] In response to a neural network model calling operation performed by a target user, the target calling configuration file is loaded to complete the loading of multiple neural network models, and a target industrial camera is triggered to start up to perform an image acquisition operation to obtain corresponding image data to be processed;

[0009] The image data to be processed is loaded into the multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return corresponding image data processing results, wherein the data loaded into the first neural network model among the multiple neural network models is the image data to be processed, and the data loaded into each neural network model other than the first neural network model among the multiple neural network models is the image data processing result returned by the previous neural network model.

[0010] In a preferred embodiment of the present application, in the above-mentioned multi-neural network model calling method, the step of performing calling configuration processing on the multiple neural network models to be called in response to the multi-neural network model configuration operation performed by the target user to form a corresponding target calling configuration file includes:

[0011] In response to the first multiple neural network model configuration operation performed by the target user, algorithm parameter configuration processing is performed on the multiple neural network models that need to be called to form a corresponding algorithm parameter configuration file, wherein the algorithm parameter configuration file at least includes a threshold parameter, and the threshold parameter is used to reflect the amount of image data to be processed input into the multiple neural network models within a target time length;

[0012] In response to the second multiple neural network model configuration operation performed by the target user, model configuration processing is performed on the multiple neural network models that need to be called to form a corresponding model configuration file, wherein the model configuration file at least has a calling sequence of the multiple neural network models that need to be called and a calling path of each neural network model;

[0013] In response to the third multi-neural network model configuration operation performed by the target user, the startup parameters of the industrial cameras corresponding to the multiple neural network models that need to be called are configured and processed to form a corresponding camera parameter configuration file, wherein the target call configuration file includes the algorithm parameter configuration file, the model configuration file and the camera parameter configuration file.

[0014] In a preferred embodiment of the present application, in the above-mentioned multi-neural network model calling method, the step of loading the image data to be processed into the multiple neural network models so that the multiple neural network models process the image data to be processed in sequence and return the corresponding image data processing results includes:

[0015] Determine a neural network model to be called first based on a first calling parameter in the target calling configuration file, wherein the first calling parameter is used to reflect the calling sequence of each neural network model in the multiple neural network models;

[0016] Determining a calling path of the first called neural network model based on a second calling parameter in the target calling configuration file;

[0017] Based on the calling path of the first called neural network model, calling the neural network model so that the neural network model is loaded into the image data to be processed, so as to process the image data to be processed and return the corresponding image data processing result;

[0018] Determine a neural network model for a second call based on a first call parameter in the target call configuration file, and determine a call path for the neural network model for the second call based on a second call parameter in the target call configuration file;

[0019] Based on the calling path of the second called neural network model, the neural network model is called so that the neural network model is loaded into the image data processing result returned by the first called neural network model to process the image data processing result and return the corresponding image data processing result.

[0020] In a preferred embodiment of the present application, in the above-mentioned multi-neural network model calling method, the step of determining the second called neural network model based on the first calling parameter in the target calling configuration file, and determining the calling path of the second called neural network model based on the second calling parameter in the target calling configuration file includes:

[0021] Based on the third call parameter in the target call configuration file, determining whether the neural network model called first belongs to the neural network model called last;

[0022] When the first called neural network model does not belong to the last called neural network model, the second called neural network model is determined based on the first calling parameter in the target calling configuration file, and the calling path of the second called neural network model is determined based on the second calling parameter in the target calling configuration file.

[0023] In a preferred embodiment of the present application, in the above-mentioned multi-neural network model calling method, the step of loading the image data to be processed into the multiple neural network models so that the multiple neural network models process the image data to be processed in sequence and return the corresponding image data processing results also includes:

[0024] Based on the third call parameter in the target call configuration file, determining whether the neural network model called secondly belongs to the neural network model called lastly;

[0025] When the second neural network model called does not belong to the last neural network model called, based on the first calling parameter in the target calling configuration file, determining the third neural network model called, and based on the second calling parameter in the target calling configuration file, determining the calling path of the third neural network model called;

[0026] Based on the calling path of the third called neural network model, the neural network model is called so that the neural network model is loaded with the image data processing result returned by the second called neural network model to process the image data processing result and return the corresponding image data processing result;

[0027] Based on the third call parameter in the target call configuration file, determining whether the neural network model called thirdly belongs to the neural network model called lastly;

[0028] When the third called neural network model belongs to the last called neural network model, the final output image data processing result is generated based on the image data processing result returned by the third called neural network model.

[0029] In a preferred embodiment of the present application, in the above-mentioned multi-neural network model calling method, when the third called neural network model belongs to the last called neural network model, the step of generating the final output image data processing result based on the image data processing result returned by the third called neural network model includes:

[0030] When the third called neural network model belongs to the last called neural network model, based on the image data processing result returned by the first called neural network model, the image data processing result returned by the second called neural network model and the image data processing result returned by the third called neural network model, the image data to be processed is annotated to form corresponding target annotated image data, wherein the target annotated image data has information of the image data processing results returned by each called neural network model;

[0031] The target labeled image data is used as the final output image data processing result.

[0032] The present application also provides a multi-neural network model calling device, which is applied to industrial cameras. The multi-neural network model calling device includes:

[0033] A model configuration module, for responding to a multi-neural network model configuration operation performed by a target user, performing call configuration processing on multiple neural network models that need to be called, and forming a corresponding target call configuration file, wherein the target call configuration file at least has a call sequence of multiple neural network models that need to be called and a call path of each neural network model;

[0034] A model calling module is used to respond to a neural network model calling operation performed by a target user, load the target calling configuration file to complete the loading of multiple neural network models, and trigger the start of a target industrial camera to perform an image acquisition operation to obtain corresponding image data to be processed;

[0035] An image data processing module is used to load the image data to be processed into the multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return corresponding image data processing results, wherein the data loaded into the first neural network model among the multiple neural network models is the image data to be processed, and the data loaded into each neural network model other than the first neural network model among the multiple neural network models is the image data processing result returned by the previous neural network model.

[0036] In a preferred embodiment of the present application, in the above-mentioned multi-neural network model calling device, the model configuration module is specifically used for:

[0037] In response to the first multiple neural network model configuration operation performed by the target user, algorithm parameter configuration processing is performed on the multiple neural network models that need to be called to form a corresponding algorithm parameter configuration file, wherein the algorithm parameter configuration file at least includes a threshold parameter, and the threshold parameter is used to reflect the amount of image data to be processed input into the multiple neural network models within a target time length;

[0038] In response to the second multiple neural network model configuration operation performed by the target user, model configuration processing is performed on the multiple neural network models that need to be called to form a corresponding model configuration file, wherein the model configuration file at least has a calling sequence of the multiple neural network models that need to be called and a calling path of each neural network model;

[0039] In response to the third multi-neural network model configuration operation performed by the target user, the startup parameters of the industrial cameras corresponding to the multiple neural network models that need to be called are configured and processed to form a corresponding camera parameter configuration file, wherein the target call configuration file includes the algorithm parameter configuration file, the model configuration file and the camera parameter configuration file.

[0040] On the basis of the above, the present application also provides an industrial camera, including:

[0041] Memory for storing computer programs;

[0042] A processor connected to the memory is used to execute the computer program stored in the memory to implement the above-mentioned multi-neural network model calling method.

[0043] Based on the above, the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is run, the various steps of the above-mentioned multi-neural network model calling method are executed.

[0044] The multi-neural network model calling method and device, industrial camera and storage medium provided by the present application, first, in response to the multi-neural network model configuration operation, the multiple neural network models to be called are called and configured to form a target calling configuration file; secondly, in response to the neural network model calling operation, the target calling configuration file is loaded to complete the loading of multiple neural network models, and the target industrial camera is triggered to start to perform image acquisition operation to obtain the image data to be processed; then, the image data to be processed is loaded into multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return the image data processing result. Based on the above content, since the target calling configuration file formed by the pre-performed multi-neural network model configuration operation at least has the calling sequence of the multiple neural network models to be called and the calling path of each neural network model, the corresponding multi-model loading can be directly performed based on the target calling configuration file in subsequent applications. Therefore, it can be directly implemented in one application without combining through multiple applications, thereby improving the problem of high complexity of the implementation logic of the multi-neural network model call existing in the prior art. In addition, when it is necessary to add or reduce neural network models, it is only necessary to adaptively adjust the target call configuration file without having to form a corresponding application (software) separately, which effectively reduces the complexity of the implementation logic of multi-model calls. In this way, the development cost and deployment cost of the application can be fully reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0046] Figure 1 This is a structural block diagram of an industrial camera provided in an embodiment of the present application.

[0047] Figure 2 A flowchart of a multi-neural network model calling method provided in an embodiment of the present application.

[0048] Figure 3Schematic diagram of the data processing relationship of multiple neural network models.

[0049] Figure 4 A block diagram of a multi-neural network model calling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0052] like Figure 1 As shown, an embodiment of the present application provides an industrial camera, wherein the industrial camera may include a memory, a processor, and a multi-neural network model calling device.

[0053] In detail, the memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, the memory and the processor can be electrically connected through one or more communication buses or signal lines. The multi-neural network model calling device includes at least one software function module stored in the memory in the form of software or firmware. The processor is used to execute an executable computer program stored in the memory, for example, the software function modules and computer programs included in the multi-neural network model calling device, so as to implement the multi-neural network model calling method provided in an embodiment of the present application.

[0054] Optionally, the memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0055] Furthermore, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0056] Understandably, Figure 1 The structure shown is for illustration only. The industrial camera may also include Figure 1 More or fewer components as shown, or with Figure 1 The different configurations shown, for example, may also include a communication unit for information interaction with other devices (such as a background server), and for another example, may also include a camera for performing image acquisition operations.

[0057] Combination Figure 2 The embodiment of the present application also provides a multi-neural network model calling method applicable to the above industrial camera. The method steps defined in the process related to the multi-neural network model calling method can be implemented by the industrial camera. Figure 2 The specific process shown is explained in detail.

[0058] Step S110, in response to the multi-neural network model configuration operation performed by the target user, call configuration processing is performed on the multiple neural network models that need to be called to form a corresponding target call configuration file.

[0059] In an embodiment of the present application, the industrial camera can respond to the multi-neural network model configuration operation performed by the target user, perform call configuration processing on the multiple neural network models that need to be called, and form a corresponding target call configuration file. Among them, the target call configuration file at least has the call sequence of the multiple neural network models that need to be called and the call path of each neural network model, based on which, multiple neural networks can be reliably called.

[0060] Step S120, in response to the neural network model calling operation performed by the target user, load the target calling configuration file to complete the loading of multiple neural network models, and trigger the target industrial camera to start up to perform image acquisition operations to obtain corresponding image data to be processed.

[0061] In an embodiment of the present application, the industrial camera can respond to the neural network model calling operation performed by the target user, load the target calling configuration file, complete the loading of multiple neural network models, and trigger the target industrial camera to start up to perform image acquisition operations to obtain corresponding image data to be processed. Specifically, the neural network model to be called can be initialized so that the image data to be processed can be processed in subsequent steps.

[0062] Step S130, loading the image data to be processed into the multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return corresponding image data processing results.

[0063] In an embodiment of the present application, the industrial camera can load the image data to be processed into the multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return the corresponding image data processing results. The data loaded into the first neural network model among the multiple neural network models is the image data to be processed, and the data loaded into each neural network model other than the first neural network model among the multiple neural network models is the image data processing result returned by the previous neural network model. For example, Figure 3 , the data loaded into the second neural network model is the image data processing result returned by the first neural network model, the data loaded into the third neural network model is the image data processing result returned by the second neural network model, and the data loaded into the fourth neural network model is the image data processing result returned by the third neural network model.

[0064] Based on the above content, since the target call configuration file formed by the pre-performed multi-neural network model configuration operation at least has the calling sequence of the multiple neural network models that need to be called and the calling path of each neural network model, the corresponding multi-model loading can be directly performed based on the target call configuration file in subsequent applications. Therefore, it can be directly implemented in one application without combining multiple applications, thereby improving the high complexity of the implementation logic of multi-neural network model calls in the prior art. In addition, when it is necessary to increase or decrease the neural network model, it is only necessary to adaptively adjust the target call configuration file without separately forming the corresponding application (software), which effectively reduces the complexity of the implementation logic of multi-model calls, thereby fully reducing the development cost and deployment cost of the application.

[0065] What needs to be explained about step S110 is that the specific method of performing call configuration processing on the multiple neural network models that need to be called is not limited and can be selected according to actual needs.

[0066] For example, in an alternative implementation, in order to ensure that the calls to multiple neural network models have a higher reliability, the above-mentioned step S110 may further include step S111, step S112 and step S113, and the specific content of each step is described as follows.

[0067] Step S111, in response to the first multiple neural network model configuration operation performed by the target user, algorithm parameter configuration processing is performed on the multiple neural network models that need to be called to form a corresponding algorithm parameter configuration file.

[0068] In an embodiment of the present application, in response to the first multiple neural network model configuration operation performed by the target user, algorithm parameter configuration processing can be performed on multiple neural network models that need to be called to form a corresponding algorithm parameter configuration file (such as algorithm.config). The algorithm parameter configuration file includes at least a threshold parameter, which is used to reflect the number of image data to be processed input into the multiple neural network models within the target duration, the number of image data to be processed input into the neural network model per minute, or one frame of image data to be processed input per target duration.

[0069] Step S112, in response to the second multiple neural network model configuration operation performed by the target user, model configuration processing is performed on the multiple neural network models that need to be called to form a corresponding model configuration file.

[0070] In an embodiment of the present application, in response to the second multiple neural network model configuration operation performed by the target user, model configuration processing can be performed on the multiple neural network models that need to be called to form a corresponding model configuration file. The model configuration file (such as model.config) at least has the calling sequence of the multiple neural network models that need to be called and the calling path of each neural network model, and illustratively, it can also include information such as version, type, output size, output size, and whether it belongs to the last model.

[0071] Specifically, for the first neural network model, you can have the following information:

[0072] id: 0——indicates the first neural network model called;

[0073] version: R1——Indicates that the version is the first version (a new version will be formed after each update iteration);

[0074] type: Location——Indicates that the model type is a location model;

[0075] path: Location-V1-0 / location-modoel-name.bin.aidem——indicates the calling path;

[0076] output-shape: [1,480,480,1]——indicates the size of the output result;

[0077] iutput-shape: [1,480,480,3]——indicates the size of the loaded data;

[0078] operate-on-next-id: 1——Indicates that it does not belong to the last called model.

[0079] Specifically, for the second neural network model, we can have the following information:

[0080] id: 1 – represents the neural network model called second;

[0081] version: R1——Indicates that the version is the first version (a new version will be formed after each update iteration);

[0082] type: Segment——indicates that the model type is a segmentation model;

[0083] path: Segment-V1-0 / segment-modoel-name.bin.aidem——indicates the calling path;

[0084] output-shape: [1,480,480,1]——indicates the size of the output result;

[0085] iutput-shape: [1,480,480,3]——indicates the size of the loaded data;

[0086] operate-on-next-id: -1——Indicates the model that belongs to the last call.

[0087] Step S113, in response to the third multi-neural network model configuration operation performed by the target user, the startup parameters of the industrial cameras corresponding to the multiple neural network models that need to be called are configured and processed to form a corresponding camera parameter configuration file.

[0088] In an embodiment of the present application, in response to the third multi-neural network model configuration operation performed by the target user, the startup parameters (such as sampling frame rate and other parameters) of the industrial camera corresponding to the multiple neural network models to be called can be configured to form a corresponding camera parameter configuration file (camera.config). The target call configuration file includes the algorithm parameter configuration file, the model configuration file and the camera parameter configuration file.

[0089] It should be noted that the above-mentioned multiple neural network models can be at least two of the models such as positioning model, segmentation model, positive sample model, target detection model, classification model and OCR model.

[0090] It should be noted that for step S130, the specific method of loading the image data to be processed into the multiple neural network models is not limited and can be selected according to actual needs.

[0091] For example, in an alternative embodiment, in order to ensure that the image data to be processed can be reliably loaded into the multiple neural network models, the above-mentioned step S130 may further include step S131, step S132, step S133, step S134 and step S135, and the specific content of each step is described as follows.

[0092] Step S131, based on the first call parameter in the target call configuration file, determine the first called neural network model.

[0093] In an embodiment of the present application, the first neural network model to be called (the corresponding "id" may be "0") may be determined based on the first calling parameter (such as the aforementioned "id") in the target calling configuration file. The first calling parameter is used to reflect the calling sequence of each neural network model in the multiple neural network models.

[0094] Step S132, determining the calling path of the first called neural network model based on the second calling parameter in the target calling configuration file.

[0095] In an embodiment of the present application, the calling path of the first called neural network model can be determined based on the second calling parameter (such as the aforementioned "path") in the target calling configuration file.

[0096] Step S133, based on the calling path of the first called neural network model, calling the neural network model, so that the neural network model is loaded into the image data to be processed, so as to process the image data to be processed and return the corresponding image data processing result.

[0097] In an embodiment of the present application, the neural network model can be called based on the calling path of the first called neural network model, so that the neural network model is loaded into the image data to be processed to process the image data to be processed and return the corresponding image data processing result. For example, the first called neural network model can be used to perform positioning processing on the image data to be processed. It should be noted that before the image data to be processed is loaded into the first called neural network model, the image data to be processed can also be preprocessed (or pre-processed), such as: image enhancement: using filtering, contrast adjustment and other methods to improve image quality; scaling and cropping: adjusting the image size and removing unnecessary parts so that the size of the input model meets the requirements; normalization: adjusting the pixel value to a specific range (such as 0 to 1), etc.

[0098] Step S134, based on the first calling parameter in the target calling configuration file, determine the neural network model for the second calling, and based on the second calling parameter in the target calling configuration file, determine the calling path of the neural network model for the second calling.

[0099] In the embodiment of the present application, the neural network model of the second call can be determined based on the first call parameter in the target call configuration file (the corresponding "id" can be "1"), and the call path of the neural network model of the second call can be determined based on the second call parameter in the target call configuration file.

[0100] Step S135, based on the calling path of the second called neural network model, the neural network model is called, so that the neural network model is loaded into the image data processing result returned by the first called neural network model to process the image data processing result and return the corresponding image data processing result.

[0101] In an embodiment of the present application, the neural network model can be called based on the calling path of the second called neural network model, so that the neural network model is loaded into the image data processing result returned by the first called neural network model to process the image data processing result and return the corresponding image data processing result. It should be noted that the image data processing result returned by the first called neural network model can be directly loaded into the second called neural network model, or the image data processing result returned by the first called neural network model can be processed (such as resizing, etc.), and then the processed data is loaded into the second called neural network model for corresponding processing (such as segmentation). The specific selection and configuration can be made according to actual needs.

[0102] It can be understood that in the above step S134, in an alternative implementation, in order to further improve the reliability of the neural network model call, the above step S134 can further include the following specific implementation sub-steps:

[0103] First, based on the third call parameter in the target call configuration file (such as the aforementioned "operate-on-next-id"), it can be determined whether the neural network model called first belongs to the neural network model called last;

[0104] Secondly, when the first called neural network model does not belong to the last called neural network model (such as "operate-on-next-id" is 1), the second called neural network model is determined based on the first calling parameter in the target calling configuration file, and the calling path of the second called neural network model is determined based on the second calling parameter in the target calling configuration file; in addition, when the first called neural network model belongs to the last called neural network model (such as "operate-on-next-id" is -1), the subsequent calling of the neural network model can be stopped.

[0105] It is understandable that, in an alternative implementation, after executing the above step S135, the above step S130 may further include the following steps:

[0106] First, based on the third call parameter in the target call configuration file, it can be determined whether the neural network model called secondly belongs to the neural network model called lastly;

[0107] Secondly, when the neural network model called secondly does not belong to the neural network model called lastly, the neural network model called thirdly can be determined based on the first calling parameter in the target calling configuration file, and the calling path of the neural network model called thirdly can be determined based on the second calling parameter in the target calling configuration file;

[0108] Then, based on the calling path of the third called neural network model, the neural network model can be called so that the neural network model is loaded into the image data processing result returned by the second called neural network model to process the image data processing result and return the corresponding image data processing result;

[0109] Further, it can be determined whether the neural network model called thirdly belongs to the neural network model called lastly based on the third calling parameter in the target calling configuration file;

[0110] Finally, when the third called neural network model belongs to the last called neural network model, the final output image data processing result can be generated based on the image data processing result returned by the third called neural network model.

[0111] It is understandable that, in an alternative embodiment, in order to ensure the reliability of the image data processing result finally outputted, the corresponding output may be performed based on the following sub-steps:

[0112] First, when the third called neural network model belongs to the last called neural network model, the image data to be processed can be annotated based on the image data processing result returned by the first called neural network model, the image data processing result returned by the second called neural network model and the image data processing result returned by the third called neural network model to form corresponding target annotated image data, wherein the target annotated image data has information on the image data processing results returned by each called neural network model;

[0113] Secondly, the target annotated image data is used as the final output image data processing result.

[0114] Based on the above multi-neural network model calling method, in a specific application scenario, the following contents may be included:

[0115] Model loading: Each time the application software (program) is started, the first step is to load the model.config file. By reading the number of models and model types in the file, the init method of the corresponding model is called. The order of model use is based on the id, that is, id=0 is the first model used for data operation, and [operate_on_next_id] determines whether it is the last model. In this application scenario, the first model is positioning, and the second model is segmentation (that is, the last model);

[0116] Data processing:

[0117] a. After the model is loaded successfully, the industrial camera is triggered to open, the acquired image data is sent to the pre-processing implementation of positioning, the pre-processing result is sent to the model inference entrance, and finally the inference result is post-processed to obtain the result map of the positioning model (image data processing result);

[0118] b. The pre-order result graph generated by the positioning model is used as the input graph of the subsequent segmentation model, and is sent to the pre-processing of the segmentation. After processing, it is sent to the segmentation model reasoning entrance, and finally the result graph of the segmentation model is obtained;

[0119] Result restoration (multiple models need to restore their respective result graphs to the original graph):

[0120] a. Put the segmented result image back to the same position in the original image according to the clipping coordinates obtained when the positioning result is output;

[0121] b. Use the coordinates generated by the positioning reasoning results to mark the positioning area in the original image.

[0122] Based on the above steps, you can achieve:

[0123] Adaptive model combination: Generate applications based on the model combination of the model training scenario. The template is unique and there is no need to repeatedly implement business logic.

[0124] Dynamic data flow: the result graph of the previous model is used as the input graph of the subsequent model;

[0125] Result graph restoration: Multiple models are restored in reverse, starting from the last model and ending with the first model, and finally restored to the original graph, putting the results generated by all models into the original graph.

[0126] Combination Figure 4 The embodiment of the present application also provides a multi-neural network model calling device applicable to the above industrial camera. The multi-neural network model calling device may include a model configuration module, a model calling module and an image data processing module.

[0127] The model configuration module is used to respond to the multi-neural network model configuration operation performed by the target user, perform call configuration processing on the multiple neural network models that need to be called, and form a corresponding target call configuration file, wherein the target call configuration file at least has the call sequence of the multiple neural network models that need to be called and the call path of each neural network model. In the embodiment of the present application, the model configuration module can be used to execute Figure 2 As shown in step S110, the relevant contents of the model configuration module can refer to the above description of step S110.

[0128] The model calling module is used to respond to the neural network model calling operation performed by the target user, load the target calling configuration file to complete the loading of multiple neural network models, and trigger the target industrial camera to start up to perform image acquisition operations to obtain corresponding image data to be processed. In the embodiment of the present application, the model calling module can be used to execute Figure 2 As shown in step S120, for the relevant content of the model calling module, please refer to the above description of step S120.

[0129] The image data processing module is used to load the image data to be processed into the multiple neural network models, so that the multiple neural network models process the image data to be processed in turn and return the corresponding image data processing results, wherein the data loaded into the first neural network model among the multiple neural network models is the image data to be processed, and the data loaded into each neural network model other than the first neural network model among the multiple neural network models is the image data processing result returned by the previous neural network model. In the embodiment of the present application, the image data processing module can be used to perform Figure 2 As shown in step S130, the relevant contents of the image data processing module can refer to the above description of step S130.

[0130] It is understandable that, in an alternative implementation, the model configuration module may be specifically used for:

[0131] First, in response to a first multi-neural network model configuration operation performed by a target user, algorithm parameter configuration processing can be performed on multiple neural network models that need to be called to form a corresponding algorithm parameter configuration file, wherein the algorithm parameter configuration file at least includes a threshold parameter, and the threshold parameter is used to reflect the number of to-be-processed image data input into the multiple neural network models within a target time length; secondly, in response to a second multi-neural network model configuration operation performed by the target user, model configuration processing can be performed on multiple neural network models that need to be called to form a corresponding model configuration file, wherein the model configuration file at least has a calling sequence of the multiple neural network models that need to be called and a calling path of each neural network model; then, in response to a third multi-neural network model configuration operation performed by the target user, startup parameters of industrial cameras corresponding to the multiple neural network models that need to be called can be configured to form a corresponding camera parameter configuration file, wherein the target call configuration file includes the algorithm parameter configuration file, the model configuration file and the camera parameter configuration file.

[0132] In an embodiment of the present application, corresponding to the above-mentioned multi-neural network model calling method applied to the industrial camera, a computer-readable storage medium is also provided, in which a computer program is stored, and the computer program executes each step of the multi-neural network model calling method when it is run. Among them, the steps executed when the aforementioned computer program is run will not be repeated here one by one, and reference can be made to the explanation of the multi-neural network model calling method in the previous text.

[0133] In summary, the multi-neural network model calling method and device, industrial camera and storage medium provided by the present application, first, in response to the multi-neural network model configuration operation, call configuration processing is performed on the multiple neural network models that need to be called to form a target call configuration file; secondly, in response to the neural network model calling operation, the target call configuration file is loaded to complete the loading of multiple neural network models, and trigger the target industrial camera to start to perform image acquisition operation to obtain the image data to be processed; then, the image data to be processed is loaded into multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return the image data processing result. Based on the above content, since the target call configuration file formed by the pre-performed multi-neural network model configuration operation at least has the call sequence of the multiple neural network models that need to be called and the call path of each neural network model, the corresponding multi-model loading can be directly performed based on the target call configuration file in subsequent applications. Therefore, it can be directly implemented in one application without combining through multiple applications, thereby improving the problem of high complexity of the implementation logic of the multi-neural network model call in the prior art. In addition, when it is necessary to add or reduce neural network models, it is only necessary to adaptively adjust the target call configuration file without having to form a corresponding application (software) separately, which effectively reduces the complexity of the implementation logic of multi-model calls. In this way, the development cost and deployment cost of the application can be fully reduced.

[0134] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device and method embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0136] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0137] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for calling a multi-neural network model, characterized in that: Applied to industrial cameras, the multi-neural network model calling method includes: In response to the multi-neural network model configuration operation performed by the target user, a call configuration process is performed on the multiple neural network models that need to be called to form a corresponding target call configuration file, wherein the target call configuration file at least has a call sequence of the multiple neural network models that need to be called and a call path of each neural network model; In response to a neural network model calling operation performed by a target user, the target calling configuration file is loaded to complete the loading of multiple neural network models, and a target industrial camera is triggered to start up to perform an image acquisition operation to obtain corresponding image data to be processed; Loading the image data to be processed into the multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return corresponding image data processing results, wherein the data loaded into the first neural network model among the multiple neural network models is the image data to be processed, and the data loaded into each neural network model other than the first neural network model among the multiple neural network models is the image data processing result returned by the previous neural network model; The step of performing call configuration processing on multiple neural network models to be called in response to the multi-neural network model configuration operation performed by the target user to form a corresponding target call configuration file includes: In response to the first multiple neural network model configuration operation performed by the target user, algorithm parameter configuration processing is performed on the multiple neural network models that need to be called to form a corresponding algorithm parameter configuration file, wherein the algorithm parameter configuration file at least includes a threshold parameter, and the threshold parameter is used to reflect the amount of image data to be processed input into the multiple neural network models within a target time length; In response to the second multiple neural network model configuration operation performed by the target user, model configuration processing is performed on the multiple neural network models that need to be called to form a corresponding model configuration file, wherein the model configuration file at least has a calling sequence of the multiple neural network models that need to be called and a calling path of each neural network model; In response to the third multi-neural network model configuration operation performed by the target user, the startup parameters of the industrial cameras corresponding to the multiple neural network models that need to be called are configured and processed to form a corresponding camera parameter configuration file, wherein the target call configuration file includes the algorithm parameter configuration file, the model configuration file and the camera parameter configuration file.

2. The multi-neural network model calling method according to claim 1, characterized in that: The step of loading the image data to be processed into the multiple neural network models so that the multiple neural network models process the image data to be processed in sequence and return corresponding image data processing results comprises: Determine a neural network model to be called first based on a first calling parameter in the target calling configuration file, wherein the first calling parameter is used to reflect the calling sequence of each neural network model in the multiple neural network models; Determining a calling path of the first called neural network model based on a second calling parameter in the target calling configuration file; Based on the calling path of the first called neural network model, calling the neural network model so that the neural network model is loaded into the image data to be processed, so as to process the image data to be processed and return the corresponding image data processing result; Determine a neural network model for a second call based on a first call parameter in the target call configuration file, and determine a call path for the neural network model for the second call based on a second call parameter in the target call configuration file; Based on the calling path of the second called neural network model, the neural network model is called so that the neural network model is loaded into the image data processing result returned by the first called neural network model to process the image data processing result and return the corresponding image data processing result.

3. The multi-neural network model calling method according to claim 2, characterized in that: The step of determining the neural network model for the second call based on the first call parameter in the target call configuration file, and determining the call path of the neural network model for the second call based on the second call parameter in the target call configuration file includes: Based on the third call parameter in the target call configuration file, determining whether the neural network model called first belongs to the neural network model called last; When the first called neural network model does not belong to the last called neural network model, the second called neural network model is determined based on the first calling parameter in the target calling configuration file, and the calling path of the second called neural network model is determined based on the second calling parameter in the target calling configuration file.

4. The multi-neural network model calling method according to claim 2, characterized in that: The step of loading the image data to be processed into the multiple neural network models so that the multiple neural network models process the image data to be processed in sequence and return corresponding image data processing results also includes: Based on the third call parameter in the target call configuration file, determining whether the neural network model called secondly belongs to the neural network model called lastly; When the second neural network model called does not belong to the last neural network model called, based on the first calling parameter in the target calling configuration file, determining the third neural network model called, and based on the second calling parameter in the target calling configuration file, determining the calling path of the third neural network model called; Based on the calling path of the third called neural network model, the neural network model is called so that the neural network model is loaded with the image data processing result returned by the second called neural network model to process the image data processing result and return the corresponding image data processing result; Based on the third call parameter in the target call configuration file, determining whether the neural network model called thirdly belongs to the neural network model called lastly; When the third called neural network model belongs to the last called neural network model, the final output image data processing result is generated based on the image data processing result returned by the third called neural network model.

5. The multi-neural network model calling method according to claim 4, characterized in that: The step of generating a final output image data processing result based on the image data processing result returned by the third called neural network model when the third called neural network model belongs to the last called neural network model includes: When the third called neural network model belongs to the last called neural network model, based on the image data processing result returned by the first called neural network model, the image data processing result returned by the second called neural network model and the image data processing result returned by the third called neural network model, the image data to be processed is annotated to form corresponding target annotated image data, wherein the target annotated image data has information of the image data processing results returned by each called neural network model; The target labeled image data is used as the final output image data processing result.

6. A multi-neural network model calling device, characterized in that: Applied to industrial cameras, the multi-neural network model calling device includes: A model configuration module, for responding to a multi-neural network model configuration operation performed by a target user, performing call configuration processing on multiple neural network models that need to be called, and forming a corresponding target call configuration file, wherein the target call configuration file at least has a call sequence of multiple neural network models that need to be called and a call path of each neural network model; A model calling module is used to respond to a neural network model calling operation performed by a target user, load the target calling configuration file to complete the loading of multiple neural network models, and trigger the start of a target industrial camera to perform an image acquisition operation to obtain corresponding image data to be processed; An image data processing module, used for loading the image data to be processed into the multiple neural network models, so that the multiple neural network models process the image data to be processed in sequence and return corresponding image data processing results, wherein the data loaded into the first neural network model among the multiple neural network models is the image data to be processed, and the data loaded into each neural network model other than the first neural network model among the multiple neural network models is the image data processing result returned by the previous neural network model; Wherein, the model configuration module is specifically used for: In response to the first multiple neural network model configuration operation performed by the target user, algorithm parameter configuration processing is performed on the multiple neural network models that need to be called to form a corresponding algorithm parameter configuration file, wherein the algorithm parameter configuration file at least includes a threshold parameter, and the threshold parameter is used to reflect the amount of image data to be processed input into the multiple neural network models within a target time length; In response to the second multiple neural network model configuration operation performed by the target user, model configuration processing is performed on the multiple neural network models that need to be called to form a corresponding model configuration file, wherein the model configuration file at least has a calling sequence of the multiple neural network models that need to be called and a calling path of each neural network model; In response to the third multi-neural network model configuration operation performed by the target user, the startup parameters of the industrial cameras corresponding to the multiple neural network models that need to be called are configured and processed to form a corresponding camera parameter configuration file, wherein the target call configuration file includes the algorithm parameter configuration file, the model configuration file and the camera parameter configuration file.

7. An industrial camera, characterized in that: include: Memory for storing computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the multi-neural network model calling method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when running, executes the multi-neural network model calling method described in any one of claims 1 to 5.

Images