Data loading method and device, electronic equipment and storage medium

Through the producer-consumer model, the problem of resource consumption and time extension during large-scale data set import in MBD software is solved, and efficient data loading is achieved.

CN118885232APending Publication Date: 2024-11-01CHONGQING SELIS PHOENIX INTELLIGENT INNOVATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410959594.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

In the MBD software development process, when importing large-scale data sets, the existing technology leads to increased CPU and memory resource consumption, and the data loading time is extended, which affects development efficiency.

Method used

Using the producer-consumer model, data is read concurrently through multiple producer coroutines and piped to the consumer coroutines, reducing resource consumption and loading time.

Benefits of technology

It improves the efficiency of data loading, reduces resource consumption and loading time, and improves the efficiency of software development.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118885232A_ABST
    Figure CN118885232A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data loading, and discloses a data loading method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining to-be-loaded data, storing the to-be-loaded data to a preset file, obtaining a plurality of producer coroutines and consumer coroutines for data loading based on preset producer-consumer model configuration, and loading the data to the consumer coroutines through the producer coroutines and the consumer coroutines. Concurrently reading to-be-loaded data in a preset file through a plurality of producer coroutines, concurrently transmitting the to-be-loaded data to a plurality of consumer coroutines through a pipeline, and transmitting the to-be-loaded data to a target working area in response to the to-be-loaded data concurrently transmitted by the plurality of consumer coroutines so as to complete data loading of the to-be-loaded data; through the method, a more practical and quicker data loading method is provided, so that resource consumption in the data loading process is reduced, and the data loading time is shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of data loading, and in particular to a data loading method, device, electronic device and storage medium. Background Art

[0002] In the MBD (Model-Based Design) software development process, in order to achieve specific system behaviors, each functional module will define a series of signal quantities, calibration parameters, and constants in detail based on the content of the DBC (CAN Data Base CANoe, which is the database file of the CAN network, usually used to define signals, messages, etc. in the CAN communication protocol) file. These parameters are the basic elements in the system design and simulation process. They are used to build system models and perform simulation analysis in the Simulink environment. Usually, these parameters are first centrally managed and defined in an EXCEL table to facilitate data organization, editing, and sharing. Then, during the development phase of Simulink, the data defined in EXCEL needs to be imported into the MATLAB workspace so that they can be directly referenced and used in the Simulink model.

[0003] In the related implementation method, this import process is completed by writing and executing an m script (MATLAB script file) in the MATLAB environment, which reads each data item in the EXCEL file one by one and loads it into the MATLAB workspace. However, this method has significant resource consumption problems when processing large-scale data sets, including increased CPU and memory usage, and significantly extended data loading time. These unfavorable factors are particularly evident in the complex process of software development and integration, which may lead to an extension of the development cycle and a reduction in efficiency. Obviously, there is an urgent need for a new data loading method to solve at least one of the above problems.

[0004] It should be noted that the above content only provides background technical information related to the present application and does not necessarily constitute prior art. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, the present application provides a data loading method, device, electronic device and storage medium to reduce resource consumption during data loading and shorten loading time.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a data loading method is provided, including: obtaining data to be loaded, and saving the data to be loaded to a preset file; obtaining multiple producer coroutines and consumer coroutines for data loading based on a preset producer-consumer model configuration; concurrently reading the data to be loaded in the preset file by multiple producer coroutines, and concurrently transmitting the data to be loaded to multiple consumer coroutines through a pipeline; in response to the data to be loaded concurrently by multiple consumer coroutines, transmitting the data to be loaded to a target workspace, so as to complete the data loading of the data to be loaded.

[0008] In one embodiment of the present application, based on the aforementioned scheme, before obtaining multiple producer coroutines and consumer coroutines for data loading based on a preset producer-consumer model configuration, the method also includes: obtaining the file name in the folder where the preset file is located; performing keyword recognition on the file name to obtain a recognition result; determining the number of target files based on the recognition result; and determining the number of producer coroutines based on the number of target files.

[0009] In one embodiment of the present application, based on the aforementioned scheme, before obtaining multiple producer coroutines and consumer coroutines for data loading based on a preset producer-consumer model configuration, the method also includes: obtaining the total number of calibration quantities, the total number of semaphores, the total number of constants, the total memory size of the server, and the number of CPU cores of the server, wherein the data to be loaded includes semaphores, calibration quantities and constants; testing the resource occupancy of a calibration quantity task, calculating the relative consumption ratio of the calibration quantity task to the CPU and memory, and obtaining a first weight factor of the calibration quantity to resource consumption; testing the resource occupancy of a semaphore task, calculating the relative consumption ratio of the semaphore task to the CPU and memory, and obtaining a second weight factor of the semaphore to resource consumption; testing the resource occupancy of a constant task, calculating the relative consumption ratio of the constant task to the CPU and memory, and obtaining a third weight factor of the constant to resource consumption; determining the number of consumer coroutines by the total number of calibration quantities, the total number of semaphores, the total number of constants, the total memory size of the server, the number of CPU cores of the server, the first weight factor, the second weight factor and the third weight factor.

[0010] In one embodiment of the present application, based on the above solution, the number of producer coroutines is determined by the following formula: Where X represents the number of producer coroutines, n represents the number of target files, and F i is an indicator function indicating whether the i-th file is the target file. If it is the target file, then F i =1, otherwise, F i =0.

[0011] In one embodiment of the present application, based on the above solution, the number of consumer coroutines is determined by the following formula: Wherein, Y represents the number of consumer coroutines, B represents the total number of calibration quantities, S represents the total number of semaphores, K represents the total number of constants, M represents the total memory size of the server, C represents the number of CPU cores of the server, and W represents the total number of semaphores. b represents the first weight factor, W s represents the second weight factor, W k represents the third weight factor, and K0 represents a preset reference constant used to adjust the overall ratio.

[0012] In one embodiment of the present application, based on the aforementioned scheme, after obtaining the data to be loaded and saving the data to be loaded to a preset file, the method further includes: compiling a target executable file according to the data to be loaded and preset compilation rules; establishing an interface for calling the target program in the target computing programming software, and calling the target executable file to interact with the target program through the system command of the target computing programming software; transferring data between the target computing programming software and the target program through a preset transfer method to enable data interaction between the target computing programming software and the target program, wherein the preset transfer method includes one of file transfer, pipe transfer and memory sharing.

[0013] In one embodiment of the present application, based on the aforementioned scheme, after the data to be loaded is transferred to the target workspace, the method further includes: comparing the model variables of the graphical simulation tool with the data to be loaded, and verifying the integrity and accuracy of the data loading according to the comparison results; compiling and simulating the model of the data to be loaded through the graphical simulation tool, and verifying the validity and performance of the data loading according to the simulation results.

[0014] According to one aspect of an embodiment of the present application, a data loading device is provided, including: a storage module, used to obtain data to be loaded and save the data to be loaded to a preset file; a configuration module, used to obtain multiple producer coroutines and consumer coroutines for data loading based on a preset producer-consumer model configuration; a transmission module, used to concurrently read the data to be loaded in the preset file through multiple producer coroutines, and concurrently transmit the data to be loaded to multiple consumer coroutines through a pipeline; a loading module, used to respond to the data to be loaded concurrently by multiple consumer coroutines, and transmit the data to be loaded to a target workspace, so as to complete the data loading of the data to be loaded.

[0015] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements a data loading method as described in any one of the above embodiments.

[0016] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer is enabled to execute the data loading method as described in any one of the above embodiments.

[0017] The beneficial effects of the present application are as follows: the present application obtains the data to be loaded, saves the data to be loaded to a preset file, obtains multiple producer coroutines and consumer coroutines for data loading based on a preset producer-consumer model configuration, concurrently reads the data to be loaded in the preset file through multiple producer coroutines, and concurrently transmits the data to be loaded to multiple consumer coroutines through pipelines, and in response to the concurrent data to be loaded by multiple consumer coroutines, transmits the data to be loaded to a target workspace to complete the data loading of the data to be loaded; concurrently reads and transmits the data to be loaded by multiple producer coroutines and multiple consumer coroutines, thereby reducing resource consumption in the data loading process and reducing the loading time of data loading, thereby improving the efficiency of data loading.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of a data loading method shown in an exemplary embodiment of the present application;

[0022] Figure 3 is a flowchart of a data loading method shown in another exemplary embodiment of the present application;

[0023] Figure 4 is a specific flow chart of a data loading method shown in an exemplary embodiment of the present application;

[0024] Figure 5 is a block diagram of a data loading device shown in an exemplary embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0027] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present application in a schematic manner, and therefore the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0028] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0029] First of all, it should be noted that MBD (Model-Based Design) software is a widely used tool in the product development process. It guides the design, development and verification of products by establishing, simulating and analyzing models. MBD is a model-based system engineering method that defines all design, manufacturing and inspection information of products, including dimensions, tolerances, materials, processes, etc., through integrated 3D models. This method can significantly reduce the number of drawings in the traditional product development process, improve design efficiency, reduce error rates, and accelerate product launch.

[0030] The DBC file is an XML file used in the Controller Area Network (CAN) bus diagnostic tool to define the CAN bus communication protocol. It contains information such as CAN communication signals, message IDs, data lengths, cycles, etc., and is the basis for communication between electronic control units (ECUs) in the CAN bus network. The DBC file defines data flags, transmission cycles, calculation formulas, value ranges, etc., so that communication between ECUs can directly and accurately transmit data, ensuring the reliability and efficiency of the CAN bus communication protocol.

[0031] Go language (often referred to as Golang) is a statically typed, compiled, concurrent programming language with garbage collection. It aims to simplify the complexity of concurrent programming while providing efficient execution speed and powerful tool chain support. The design philosophy of Go language is "less is more". It has concise syntax, powerful standard library and efficient execution efficiency, allowing developers to quickly write reliable and easy-to-maintain code.

[0032] Simulink is a block diagram environment for multi-domain simulation and model-based design. It supports system design, simulation, automatic code generation, and continuous testing and verification of embedded systems. Among them, Simulink provides a visual modeling environment where users can build models by dragging and dropping functional modules without writing a lot of code. It also provides a rich module library, including continuous modules, discrete modules, function and table modules, mathematical modules, nonlinear modules, signal and system modules, etc. Users can choose the appropriate module for modeling according to their needs.

[0033] MATLAB (Matrix Laboratory) is a commercial mathematical software, a high-level technical computing language and interactive environment for algorithm development, data visualization, data analysis, and numerical calculation. MATLAB is widely used in many fields such as engineering computing, signal processing and communication, image processing, control system design, automobile design, financial modeling design and analysis. The main features of MATLAB include: powerful matrix operation capabilities, easy-to-use programming environment, rich function library, data visualization, integration with other programming languages ​​and software (MATLAB can be integrated with programming languages ​​such as C, C++, Java, and also supports interfaces with other software and hardware, such as Excel, Simulink, etc.), users can use MATLAB Compiler to compile MATLAB code into independent executable files or shared libraries so that it can run in an environment where MATLAB is not installed.

[0034] The Producer-Consumer Model is a classic concurrent programming model used to solve synchronization and communication problems between producers and consumers in a multi-threaded environment. In this model, producers are responsible for generating data, while consumers are responsible for processing the data. Producers and consumers are usually two or more independent threads or processes, and a mechanism is needed between them to safely transfer data while avoiding race conditions and data inconsistency problems. Among them, Producer: the thread or process responsible for generating data; Consumer: the thread or process responsible for processing (consuming) data; Buffer: the area used to temporarily store data between producers and consumers. The size of the buffer is limited, so a mechanism is needed to ensure that the producer stops producing when the buffer is full and the consumer stops consuming when the buffer is empty.

[0035] Figure 1 It is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application.

[0036] Reference Figure 1 As shown, the system architecture may include a data acquisition device 101 and a computer device 102. Among them, the computer device 102 may be at least one of a desktop graphics processing unit (Graphic Processing Unit, GPU) computer, a GPU computing cluster, a neural network computer, etc. The data acquisition device 101 is used to obtain the data to be loaded. In this embodiment, the data acquisition device 101 obtains the above data and provides it to the computer device 102 for processing. Relevant technicians can use the computer device 102 to save the data to be loaded to a preset file, obtain multiple producer coroutines and consumer coroutines for data loading based on the preset producer-consumer model configuration, read the data to be loaded in the preset file concurrently through multiple producer coroutines, and transmit the data to be loaded to multiple consumer coroutines concurrently through pipelines. In response to the concurrent data to be loaded by multiple consumer coroutines, the data to be loaded is transmitted to the target workspace to complete the data loading of the data to be loaded. It should be noted that the data acquisition device 101 and the computer device 102 provided in this embodiment are only an example, and should not bring any restrictions to the functions and scope of use of the embodiments of the present application.

[0037] It should be noted that the data loading method provided in the embodiment of the present application is generally executed by the computer device 102 , and accordingly, the data loading device is generally disposed in the computer device 102 .

[0038] Figure 2 is a flow chart of a data loading method shown in an exemplary embodiment of the present application. The data loading method can be executed by a computing and processing device. The computing and processing device can be Figure 1 The computer device 102 shown in FIG. Figure 2 As shown, the data loading method at least includes steps S210 to S240, which are described in detail as follows:

[0039] In step S210, the data to be loaded is obtained and saved to a preset file.

[0040] In one embodiment of the present application, taking the preset file as an Excel file as an example, according to the definition of DBC (CAN database file), the data to be loaded (including signal quantities, calibration quantities and constants) are recorded in the Excel file for use in a graphical simulation tool (such as a Simulink model). And the directory path containing the model development file (such as the path of the MBD folder) is added to the target workspace of the target computing programming software (such as Matlab).

[0041] In this embodiment, an Excel file is created, and an appropriate number of worksheets is set as needed. Each worksheet can be used to store different types of data (such as semaphores, calibration quantities, constants, etc.). And the data format is set, for example, a semaphore is set, and a semaphore generally includes information such as a signal name (Signal Name), ID, length (Bit Length), data type (such as Signed, Unsigned, Float, etc.), physical value range (maximum value, minimum value), precision (Resolution), offset (Offset), etc.; the calibration quantity set may include information such as a calibration name, calibration value, unit, etc., and the set constant simply records the constant name and the corresponding value. According to the above-mentioned data format, data is entered item by item in the Excel file. During the entry process, it is necessary to ensure the accuracy and completeness of the data, especially key information such as the ID, length and data type of the semaphore.

[0042] In step S220, a plurality of producer coroutines and consumer coroutines for data loading are configured based on a preset producer-consumer model.

[0043] In this embodiment, first, a pipeline is defined for transferring data between the producer coroutine and the consumer coroutine. Then, multiple producer coroutines are created, and each producer coroutine puts the data to be loaded in the preset file it reads into the pipeline. At the same time, multiple consumer coroutines are created, and each consumer coroutine takes data from the pipeline. It should be noted that this embodiment is only for illustration and does not limit the way in which the present application configures the producer coroutine and the consumer coroutine.

[0044] In one embodiment of the present application, taking the target computing programming software as Matalb and the target program as the Go program as an example, the target computing programming software calls the target program, that is, Matalb calls the Go program, and defines X producer coroutine processing functions and Y consumer coroutines for data loading in the Go program. These producer coroutines are responsible for reading data from the Excel file and transmitting it to the Y consumer coroutines through pipelines.

[0045] In one embodiment of the present application, a target executable file is compiled according to the data to be loaded and preset compilation rules; an interface for calling the target program is established in the target computing programming software, and the target executable file is called to interact with the target program through the system command of the target computing programming software; data between the target computing programming software and the target program is transferred through a preset transfer method to enable data interaction between the target computing programming software and the target program, wherein the preset transfer method includes one of file transfer, pipe transfer and memory sharing.

[0046] In this embodiment, taking the target computing programming software as Matlab, the target executable file as Go executable file, and the target program as Go program as an example, the Go executable file is compiled according to the data to be loaded and the preset compilation rules; an interface for calling the Go program is established in Matlab, and the Go executable file is called through the Matlab system command to interact with the Go program; data between Matlab and Go program is transferred through a preset transfer method to enable data interaction between Matlab and Go program, wherein the preset transfer method includes file transfer, pipe transfer or memory sharing.

[0047] In this embodiment, the interface implementation method when Matlab calls the Go program can be specifically described as follows: Implementing the interface for calling the Go program in Matlab, the compiled Go executable file can be called through Matlab system commands (such as system or ! command) to interact with the Go program. The implementation method of data interaction can be specifically described as follows: Ensure smooth data interaction between Matlab and Go programs, and use files, pipes or memory sharing to achieve data transmission and processing.

[0048] In one embodiment of the present application, the file name in the folder where the preset file is located is obtained; keyword recognition is performed on the file name to obtain a recognition result; the number of target files is determined based on the recognition result; and the number of producer coroutines is determined based on the number of target files.

[0049] In this embodiment, the file variable definition name and folder address of the application layer simulink model can be obtained by keyword recognition of the file extension (such as *.xlsx, *.xls), for example, D:\model1\Cover_IN_xxx.xlsx, D:\model 1\Global_IN_xxx.xlsx, etc. The number of target files is obtained by counting the number of target files.

[0050] In one embodiment of the present application, the number of producer coroutines is determined by the following formula:

[0051]

[0052] In formula (1), X represents the number of producer coroutines, n represents the number of target files, and F i is an indicator function, indicating whether the i-th file is a target file. If it is a target file, then F i =1, otherwise, F i =0.

[0053] In this embodiment, X can also represent the total number of Excel files in the Simulink simulation folder, n represents the number of Excel files actually existing in the Simulink simulation folder, and F i is an indicator function indicating whether the i-th file is an Excel file. If it is an Excel file, that is, the extension includes the target keyword (e.g. .xlsx), then F i =1; otherwise F i =0.

[0054] In one embodiment of the present application, the total number of calibration quantities, the total number of semaphores, the total number of constants, the total memory size of the server, and the number of CPU cores of the server are obtained, wherein the data to be loaded includes semaphores, calibration quantities and constants; the resource occupancy of a calibration quantity task is tested, the relative consumption ratio of the calibration quantity task to the CPU and memory is calculated, and a first weight factor of the calibration quantity to resource consumption is obtained; the resource occupancy of a semaphore task is tested, the relative consumption ratio of the semaphore task to the CPU and memory is calculated, and a second weight factor of the semaphore to resource consumption is obtained; the resource occupancy of a constant task is tested, the relative consumption ratio of the constant task to the CPU and memory is calculated, and a third weight factor of the constant to resource consumption is obtained; the number of consumer coroutines is determined by the total number of calibration quantities, the total number of semaphores, the total number of constants, the total memory size of the server, the number of CPU cores of the server, the first weight factor, the second weight factor and the third weight factor.

[0055] In this embodiment, the first weight factor represents the relative consumption of system resources (CPU and memory) for processing a calibration task, which is determined by performance testing and analysis. Test the resource occupancy of the calibration task in the system and calculate its relative consumption ratio of CPU and memory; the second weight factor is the weight factor of the semaphore for resource consumption, which represents the relative consumption of system resources (CPU and memory) for processing a semaphore task, which is determined by performance testing and analysis. Test the resource occupancy of the semaphore task in the system and calculate its relative consumption ratio of CPU and memory. The third weight factor represents the relative consumption of system resources (CPU and memory) for processing a constant task, which is determined by performance testing and analysis. Test the resource occupancy of the constant task in the system and calculate its relative consumption ratio of CPU and memory. The benchmark constant is used to adjust the overall ratio so that the calculated number of consumers adapts to the actual situation of the system, which is determined by performance testing and tuning. Test based on different system configurations and workloads, and adjust the benchmark constant to find the optimal number of consumers. The above steps can determine the number of consumer coroutines, ensure the rational use of system resources and efficient processing of tasks, and achieve the optimal loading efficiency.

[0056] In one embodiment of the present application, the number of consumer coroutines is determined by the following formula:

[0057]

[0058] In formula (2), Y represents the number of consumer coroutines, B represents the total number of calibration quantities, S represents the total number of semaphores, K represents the total number of constants, M represents the total memory size of the server, C represents the number of CPU cores of the server, and W b represents the first weight factor, W s represents the second weight factor, W k represents the third weight factor, and K0 represents a preset reference constant used to adjust the overall ratio.

[0059] In step S230, the data to be loaded in the preset file is read concurrently by multiple producer coroutines, and the data to be loaded is concurrently transmitted to multiple consumer coroutines through pipelines.

[0060] In one embodiment of the present application, a producer-consumer model is used to implement concurrent loading of data. The producer coroutine is responsible for reading the data in the Excel file and transmitting it to the consumer coroutine through a pipeline.

[0061] In this embodiment, in the Go program, the producer coroutine reads the data block in the Excel file, i.e., the data to be loaded. The producer coroutine sends the data block to the consumer coroutine through a pipeline. The Go language channel is used as a pipeline to ensure efficient and secure data transmission between the producer and the consumer.

[0062] In step S240 , in response to the data to be loaded concurrently by multiple consumer coroutines, the data to be loaded is transferred to the target workspace to complete the data loading of the data to be loaded.

[0063] In one embodiment of the present application, the consumer coroutine receives data and passes it to the target workspace (e.g., Matlab workspace). In the Go program, the consumer coroutine receives a data block from a pipeline. The consumer coroutine passes the data block to Matlab by writing an m file, runs the m file, loads the variables, i.e., the data to be loaded into the workspace, and is used for subsequent graphical simulation tools (e.g., simulink) simulation.

[0064] In one embodiment of the present application, after the data to be loaded is transferred to the target workspace of the target computing programming software (such as Matlab), the model variables of the graphical simulation tool (such as Simulink) are compared with the data to be loaded, and the integrity and accuracy of the data loading are verified based on the comparison results; the model of the data to be loaded is compiled and simulated through the graphical simulation tool (such as Simulink), and the validity and performance of the data loading are verified based on the simulation results.

[0065] In the present embodiment, after the data loading is completed, the data in the Matlab workspace is verified for integrity and accuracy to ensure that all data are correctly loaded. Simulink is used to compile and simulate the loaded data to verify the validity and performance of the data loading scheme. After the Simulink model is developed, it is necessary to compile and debug to ensure the correctness and reliability of the model. The compilation process converts the Simulink model into executable code or simulation model to run in the target environment. The debugging process is used to find and solve problems in the model, including logical errors, improper parameter settings, etc. By constantly compiling and debugging, the model can be gradually optimized and improved until the design requirements are met.

[0066] In this embodiment, in Simulink, the integrity and accuracy verification between the model variables and the data to be loaded is an important step, which ensures the correct operation of the model and the reliability of the simulation results. The following is an example of a detailed description of this process: First, understand Simulink model variables. The variables in the Simulink model are an important part of the model and are used to store and transfer data. These variables can be different types such as scalars, vectors, matrices or structures, and can be used in different parts of the model (such as modules, subsystems and model levels). Simulink provides an independent workspace (Model Workspace) for each model to store and manage variables related to the model. Second, preparation of data to be loaded. The data to be loaded usually comes from external files (such as MAT files, CSV files, etc.) or variables in the MATLAB workspace. Before loading data into the Simulink model, it is necessary to ensure that the format, type, range and constraints of the data meet the requirements of the model. Three, data integrity and accuracy verification 1. Data type check. Use MATLAB function: Use the data type check function (such as isnumeric, ischar, etc.) provided by MATLAB to verify whether the data type of the data to be loaded is consistent with the type required by the model variable. 2. Data range and boundary check. Verify data range: Make sure that the data to be loaded is within the expected range or boundary. You can use MATLAB's isfinite, max, min and other functions to check whether the data contains infinity (Inf), non-numeric values ​​(NaN), or values ​​outside the valid range. 3. Data format check. Date and time data: If the data to be loaded contains date and time information, you can use the datenum function to verify whether it is a valid date and time value. Regular expression: For data that needs to conform to a specific format (such as string data), you can use MATLAB's regexp function to check whether the data matches the regular expression pattern. 4. Variable name and scope check. Variable name consistency: Make sure that the variable name of the data to be loaded is consistent with the variable name defined in the Simulink model. Scope check: Understand the scope of the variable in the Simulink model (module level, subsystem level or model level) and ensure that the data is loaded to the correct location. 5. Data loading and verification process. Use Simulink's Data Import / Export function: You can load data directly from the MATLAB workspace or external files into the Simulink model. Write custom scripts: If more complex loading and validation logic is required, you can write MATLAB scripts to handle the data loading and validation process. Use Simulink Model Callbacks: You can add callback functions to execute data validation logic during model loading, initialization, simulation start, etc.Through the above steps, the integrity and accuracy between the model variables defined in the Simulink model and the data to be loaded can be ensured, thereby improving the reliability of the model and the accuracy of the simulation results.

[0067] In this embodiment, the detailed process of using Simulink to compile and simulate the model to verify the validity and performance of the data to be loaded is described as follows: Ensure that the data format of the data to be loaded is correct and matches the requirements of the Simulink model, including the type, size, sampling rate, etc. of the data. Create or open the corresponding model in Simulink, which should be able to receive input data and generate output for subsequent analysis. Connect the data to be loaded to the input end of the model and configure the parameters to ensure that the configuration of the data loading module (such as data format, sampling time, etc.) is consistent with the data itself. Before compiling, use Simulink's model checking function to identify and correct potential errors or problems. Compile the model again and prepare the model for simulation. After the model is compiled correctly, Simulink will start the simulation process. During this process, the model will calculate the output based on the input data; observe the output of the model through Simulink's Scope or other visualization tools to understand the behavior of the system under different inputs. Analyze the simulation output data to verify whether it meets expectations. For example, check whether the output data is within a reasonable range and whether it is consistent with theoretical or experimental data. Evaluate the performance of the model simulation process, including simulation time, resource consumption, etc. If the simulation takes too long or consumes too many resources, you may need to consider optimizing the model or adjusting the simulation settings. Check if any errors or exceptions occur during the simulation. If so, analyze the cause of the error and correct it. Optimize the model based on the simulation results and performance evaluation. For example, adjust model parameters, simplify the model structure, use a more efficient solver, etc. Use the optimized model for iterative simulation to further verify the effectiveness and performance of data loading. Record the key data and observations during the simulation in detail. Write a report based on the simulation results, summarize the effectiveness of data loading and model performance, and make suggestions for improvement. Through the above steps, you can use Simulink to compile and simulate the model of the data to be loaded, and verify the effectiveness and performance of data loading based on the simulation results.

[0068] In this embodiment, the following points need to be noted: 1. Data consistency, ensure that the data in Excel is consistent with the definition in the DBC file, especially key information such as signal name, ID and length. 2. Data type matching, when importing Excel data in Simulink, pay attention to the matching of data types to avoid data parsing errors. 3. Test verification, after importing Excel data into the Simulink model, perform sufficient test verification to ensure the correctness of the data and the reliability of the model.

[0069] Figure 3 FIG. 1 is a flow chart of a data loading method shown in another exemplary embodiment of the present application. Figure 3 As shown, in an exemplary embodiment, the data to be loaded is obtained and loaded into an Excel file, multiple producer coroutines read the Excel file in a loop to obtain the data to be loaded in the Excel file, and transmit the data to be loaded to the consumer coroutine through a pipeline, execute Matlab commands, pass the data to be loaded to the Matlab workspace, verify the data integrity and accuracy, and compile and simulate in simulink to verify the validity and performance of data loading. The specific steps are described in detail in the above embodiment and will not be repeated here.

[0070] Figure 4 FIG. 1 is a schematic diagram of a specific flow chart of a data loading method shown in an exemplary embodiment of the present application. Figure 4 As shown, in an exemplary embodiment, Matalb calls the Go program to interact with data through the coroutine in the Go program, obtain the data to be loaded, and load the data to be loaded into the Excel file. X producer coroutines concurrently read the Excel file to obtain the data to be loaded in the Excel file, and transmit the data to be loaded to the consumer coroutine through the channel, i.e., the pipeline. Y consumers concurrently pass the data to be loaded to the Matlab workspace, execute Matlab commands, verify the data integrity and accuracy, and compile and simulate simulink to verify the validity and performance of data loading. The specific steps are described in detail in the above embodiment and will not be repeated here.

[0071] The present application obtains the data to be loaded, saves the data to be loaded to a preset file, configures multiple producer coroutines and consumer coroutines for data loading based on the producer-consumer model, concurrently reads the data to be loaded in the preset file by multiple producer coroutines, and concurrently transmits the data to be loaded to multiple consumer coroutines through a pipeline, and multiple consumer coroutines concurrently transmit the data to be loaded to the Matlab workspace to complete the data loading of the data to be loaded; concurrently reading and transmitting the data to be loaded by multiple producer coroutines and multiple consumer coroutines reduces resource consumption in the data loading process and reduces the loading time of data loading.

[0072] This application can increase the loading speed of calibration parameters, observation parameters, and semaphore parameters of the application layer Simulink model by several times, reduce resource consumption of computers, servers, etc. during development and CI / CD (continuous integration), and improve software development efficiency and software integration efficiency.

[0073] Figure 5is a block diagram of a data loading device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown in the figure is specifically configured in the computer device 102. The device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applied.

[0074] like Figure 5 As shown, the exemplary data loading device includes: a storage module 510 , a configuration module 520 , a transmission module 530 and a loading module 540 .

[0075] Among them, the storage module 510 is used to obtain the data to be loaded and save the data to be loaded to a preset file; the configuration module 520 is used to obtain multiple producer coroutines and consumer coroutines for data loading based on the preset producer-consumer model configuration; the transmission module 530 is used to concurrently read the data to be loaded in the preset file through multiple producer coroutines, and concurrently transmit the data to be loaded to multiple consumer coroutines through pipelines; the loading module 540 is used to respond to the data to be loaded concurrently by multiple consumer coroutines, and transmit the data to be loaded to the target workspace to complete the data loading of the data to be loaded.

[0076] It should be noted that the data loading device provided in the above embodiment and the data loading method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the data loading device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0077] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the electronic device implements the data loading method provided in the above-mentioned embodiments.

[0078] Figure 6 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603, such as executing the methods provided in the above-mentioned various embodiments. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through the bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0080] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0081] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present application are executed.

[0082] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0083] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0085] Another aspect of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer, causes the computer to execute the data loading method provided in the above-mentioned embodiments. The computer-readable storage medium may be included in the electronic device described in the above-mentioned embodiments, or may exist independently without being assembled into the electronic device.

[0086] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0087] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data loading method provided in each of the above embodiments.

[0088] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0089] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0090] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A data loading method, It is characterized in that include: Obtaining the data to be loaded, and saving the data to be loaded to a preset file; Based on the preset producer-consumer model configuration, multiple producer coroutines and consumer coroutines for data loading are obtained; Concurrently reading the data to be loaded in the preset file through the multiple producer coroutines, and concurrently transmitting the data to be loaded to the multiple consumer coroutines through pipelines; In response to the data to be loaded concurrently by the multiple consumer coroutines, the data to be loaded is transmitted to the target workspace to complete the data loading of the data to be loaded.

2. The data loading method according to claim 1, It is characterized in that Before obtaining a plurality of producer coroutines and consumer coroutines for data loading based on a preset producer-consumer model configuration, the method further includes: Get the file name in the folder where the preset file is located; Perform keyword recognition on the file name to obtain a recognition result; Determine the number of target files according to the recognition result; The number of the producer coroutines is determined based on the number of the target files.

3. The data loading method according to claim 1, It is characterized in that Before obtaining a plurality of producer coroutines and consumer coroutines for data loading based on a preset producer-consumer model configuration, the method further includes: Obtain the total number of calibration quantities, the total number of semaphores, the total number of constants, the total memory size of the server, and the number of CPU cores of the server, wherein the data to be loaded includes semaphores, calibration quantities, and constants; Test the resource usage of a calibration task, calculate the relative consumption ratio of the calibration task to the CPU and memory, and obtain a first weight factor of the calibration task to the resource consumption; Testing the resource usage of a semaphore task, calculating the relative consumption ratio of the semaphore task to the CPU and the memory, and obtaining a second weight factor of the semaphore to the resource consumption; Testing the resource usage of a constant task, calculating the relative consumption ratio of the constant task to the CPU and memory, and obtaining a third weight factor of the constant to the resource consumption; The number of consumer coroutines is determined by the total number of the calibration quantities, the total number of the semaphores, the total number of the constants, the total memory size of the server, the number of CPU cores of the server, the first weight factor, the second weight factor and the third weight factor.

4. The data loading method according to claim 2, It is characterized in that The number of producer coroutines is determined by the following formula: Where X represents the number of producer coroutines, n represents the number of target files, and F i is an indicator function indicating whether the i-th file is the target file. If it is the target file, then F i =1, otherwise, F i =0.

5. The data loading method according to claim 3, It is characterized in that The number of consumer coroutines is determined by the following formula: Wherein, Y represents the number of consumer coroutines, B represents the total number of calibration quantities, S represents the total number of semaphores, K represents the total number of constants, M represents the total memory size of the server, C represents the number of CPU cores of the server, and W represents the total number of semaphores. b represents the first weight factor, W s represents the second weight factor, W k represents the third weight factor, K 0 Represents the preset base constant used to adjust the overall ratio.

6. The data loading method according to any one of claims 1 to 5, It is characterized in that After obtaining the data to be loaded and saving the data to be loaded into a preset file, the method further includes: Compiling a target executable file according to the data to be loaded and preset compilation rules; Establishing an interface for calling a target program in the target computing programming software, and calling the target executable file to interact with the target program through a system command of the target computing programming software; The data between the target computing programming software and the target program are transferred via a preset transfer method to enable data interaction between the target computing programming software and the target program, wherein the preset transfer method includes one of file transfer, pipeline transfer and memory sharing.

7. The data loading method according to any one of claims 1 to 5, It is characterized in that After transferring the data to be loaded to the target workspace, the method further includes: Comparing the model variables of the graphical simulation tool with the data to be loaded, and verifying the integrity and accuracy of the data loading according to the comparison result; The graphical simulation tool is used to compile and simulate the model of the data to be loaded, and the validity and performance of the data loading are verified according to the simulation results.

8. A data loading device, It is characterized in that include: A storage module, used for acquiring data to be loaded and saving the data to be loaded into a preset file; The configuration module is used to configure multiple producer coroutines and consumer coroutines for data loading based on the preset producer-consumer model; A transmission module, used for concurrently reading the data to be loaded in the preset file through the multiple producer coroutines, and concurrently transmitting the data to be loaded to the multiple consumer coroutines through a pipeline; The loading module is used to transmit the data to be loaded to the target workspace in response to the data to be loaded concurrently by the multiple consumer coroutines, so as to complete the data loading of the data to be loaded.

9. An electronic device, It is characterized in that The electronic device comprises: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the data loading method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the data loading method according to any one of claims 1 to 7.