Volume table comprehensive management method, system and equipment

Abstract

The invention relates to the technical field of data storage and management, and discloses a volume table comprehensive management method, system and device.The method comprises the steps that in response to a received data extraction request sent by a user through a front end, a back end executes a corresponding data extraction program according to the source file type of a volume table, and extracted table data is obtained; responding to a received data arrangement request sent by a user through a front end; in response to a received data verification request sent by a user through the front end, the back end calls a corresponding data verification model in the database according to the configuration parameters of the front end, and verifies the standard table data to obtain target table data; and in response to the received data storage request sent by the user through the front end, the back end stores the target table data into the database, so that the technical problem of poor data management effect of the volume table in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of data storage and management technology, and in particular to a method, system and device for comprehensive management of volume tables. Background Technology

[0002] As the most important component of a ship inspection certificate, the volume table is typically stored electronically as a scanned PDF, a raw PDF, or an Excel file. Current technologies generally only process volume tables in Excel format, failing to handle data from raw PDFs or scanned PDFs, resulting in limited processing options. Furthermore, for data verification, current technologies typically rely on generating charts from Excel and then visually verifying the data, leading to significant verification errors. Summary of the Invention

[0003] This application provides a method, system, and device for comprehensive management of volume tables, which solves the technical problem of poor data management of volume tables in related technologies and achieves the technical effect of accurate, efficient, and unified volume table data management.

[0004] To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a method for comprehensive management of volume tables. The method is applied to a comprehensive volume table management system, which includes a front-end, a back-end, and a database. The method includes: in response to receiving a data extraction request sent by a user through the front-end, the back-end executes a corresponding data extraction program based on the source file type of the volume table to obtain extracted table data; in response to receiving a data processing request sent by a user through the front-end, the back-end calls a corresponding table template in the database according to the front-end's configuration parameters and converts the extracted table data into standard table data according to the table template; in response to receiving a data verification request sent by a user through the front-end, the back-end calls a corresponding data verification model in the database according to the front-end's configuration parameters to verify the standard table data and obtain target table data; and in response to receiving a data storage request sent by a user through the front-end, the back-end stores the target table data in the database.

[0005] The volume table integrated management method provided in this application embodiment is applied to a volume table integrated management system, which includes a front-end, a back-end, and a database. The method includes: in response to receiving a data extraction request sent by a user through the front-end, the back-end executes a corresponding data extraction program according to the source file type of the volume table to obtain extracted table data; in response to receiving a data processing request sent by a user through the front-end, the back-end calls the corresponding table template in the database according to the front-end's configuration parameters and converts the extracted table data into standard table data according to the table template; in response to receiving a data verification request sent by a user through the front-end, the back-end calls the corresponding data verification model in the database according to the front-end's configuration parameters to verify the standard table data and obtain target table data; in response to receiving a data storage request sent by a user through the front-end, the back-end stores the target table data in the database. This method solves the technical problem of poor data management effect of volume tables in related technologies and achieves the technical effect of accurate, efficient, and unified volume table data management.

[0006] Optionally, the volume table integrated management method further includes: in response to receiving a login request sent by a user through the front end, wherein the login request includes login information, the back end verifies the login information based on user data in the database, and if the verification is successful, sends the corresponding token to the front end.

[0007] Optionally, the volume table integrated management method further includes: in response to receiving a first data query request sent by a user through the front end, the back end encapsulates the corresponding data in the database into JSON format and sends it to the front end via WebSocket.

[0008] Optionally, the volume table integrated management system further includes an external interface, and the method further includes: in response to receiving a second data query request sent by a user through the external interface, the backend encapsulates the corresponding data in the database into JSON format and sends it to the external interface via WebSocket.

[0009] Optionally, the source file type of the volume table includes native PDF, scanned PDF, and Excel. The backend executes the corresponding data extraction program according to the source file type of the volume table to obtain the extracted tabular data. Specifically, this includes: if the source file type of the volume table is native PDF, then the data is extracted using the Python tabula-py library; if the source file type of the volume table is scanned PDF, then the scanned PDF is converted into an image, and data in the image is extracted and recognized using OCR; if the source file type of the volume table is Excel format, then the data in the Excel document is read using the Python pandas, openpyxl, and xlrd libraries.

[0010] Optionally, the step of extracting and recognizing data from the image using OCR specifically includes: preprocessing the image, including excluding non-target elements, performing grayscale conversion, binarization, and morphological processing on the image, and correcting the image using affine transformation; based on the preprocessed image, summing the non-zero pixels in each row of the image, establishing a cumulative frequency histogram, finding and adding appropriate row dividing lines based on the cumulative frequency histogram, and obtaining an image divided into rows; based on the image divided into rows, extracting and recognizing the data content and position information of each text block in the image using OCR; generating a first dot matrix image based on the position information of each text block; clustering the first dot matrix image according to the size of the horizontal and vertical coordinates to generate a second dot matrix image; and obtaining the extracted tabular data according to the mapping relationship between the first and second dot matrix images.

[0011] Optionally, the step of verifying the standard table data to obtain the target table data specifically includes: using a data verification model to perform anomaly detection on the standard table data, including jump value detection, missing value detection, duplicate paragraph detection, and human tampering trace detection, to obtain anomaly detection results.

[0012] Optionally, the step of validating the standard table data to obtain the target table data further includes: the backend sending the anomaly detection result to the frontend, and the user correcting the standard table data according to the anomaly detection result to obtain the target table data.

[0013] Secondly, embodiments of this application provide a volume table integrated management system, which includes a front-end, a back-end, and a database, wherein: the front-end is used to send data extraction requests, data processing requests, data verification requests, and data storage requests to the back-end; and to configure parameters and send them to the back-end; the back-end is used to, in response to receiving a data extraction request sent by a user through the front-end, execute the corresponding data extraction program according to the source file type of the volume table to obtain extracted table data; in response to receiving a data processing request sent by a user through the front-end, call the corresponding table template in the database according to the configuration parameters of the front-end, and convert the extracted table data into standard table data according to the table template; in response to receiving a data verification request sent by a user through the front-end, call the corresponding data verification model in the database according to the configuration parameters of the front-end to verify the standard table data and obtain target table data; and in response to receiving a data storage request sent by a user through the front-end, store the target table data in the database; the database is used to store the table template, the data verification model, and the target table data.

[0014] Thirdly, embodiments of this application provide a computer device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the above-described volume table comprehensive management method.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the above-described volume table integrated management method.

[0016] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which are used to cause a computer to execute the above-described volume table integrated management method. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart of the volume table integrated management method provided in the embodiments of this application; Figure 2 A flowchart illustrating the data processing provided in the embodiments of this application; Figure 3 A flowchart for data verification provided in the embodiments of this application; Figure 4 A flowchart of the volume table integrated management method provided in the embodiments of this application; Figure 5 A schematic diagram of the volumetric meter integrated management system provided in an embodiment of this application; Figure 6 This is a schematic diagram of a data preview interface provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] As the most crucial component of ship inspection certificates, volume tables are typically stored electronically in three formats: scanned PDF, original PDF, and Excel. Scanned PDFs are obtained by scanning and uploading paper documents; each page is an image stored in bitmap format, without text, and cannot be searched, selected, or copied. Original PDFs contain text and can be searched, selected, and copied. Current technologies generally only process volume tables in Excel format, failing to handle original or scanned PDFs, resulting in limited processing capabilities. Given the numerous stakeholders involved in ship inspections and the diverse types of volume tables, current technologies can only batch process fixed types, requiring manual processing for others. Furthermore, data verification typically relies on generating charts in Excel and then visually verifying the data, leading to significant verification errors. In conclusion, current technologies are ineffective for managing volume table data, failing to provide accurate, efficient, and unified data management.

[0021] According to an embodiment of this application, a method for comprehensive management of volume tables is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0022] Please see Figure 1 , Figure 1 A flowchart illustrating the comprehensive management method for volumetric tables provided in this application embodiment. Figure 1 As shown, the above-described comprehensive management method for volume tables is applied to a comprehensive management system for volume tables, which includes a front-end, a back-end, and a database. The specific steps of the above-described comprehensive management method for volume tables are as follows: Step S1: In response to receiving a data extraction request sent by the user through the front end, the back end executes the corresponding data extraction program according to the source file type of the volume table to obtain the extracted table data; When a user needs to convert the source file of a volume table into Excel-formatted spreadsheet data, a data extraction request is sent through the front end. The source file types for volume tables include native PDFs, scanned PDFs, and Excel spreadsheets. The back end determines the source file type and executes the corresponding data extraction program. Users can click on the data preview to view an image of the data table from the source file on the front end's data display interface. Different source file types correspond to different data extraction processes. The extracted table data is editable and manipulable text data.

[0023] For example, the front end sends a data extraction request to the back end via WebSocket. The back end executes the corresponding data extraction program based on the source file type of the volume table, and sends the extraction task progress information to the front end for display in real time via WebSocket.

[0024] Step S3: In response to receiving a data processing request sent by the user through the front end, the back end calls the corresponding table template in the database according to the configuration parameters of the front end, and converts the extracted table data into standard table data according to the table template. Because the source files for volume tables vary in type, the extracted tabular data also varies in format; for example, some extracted tabular data are three-column tables, while others are six-column tables. Please refer to [link / reference]. Figure 2 , Figure 2 A flowchart for data processing provided in the embodiments of this application, such as Figure 2 As shown, when a user needs to organize extracted tabular data into a unified standard format, they configure parameters through the front-end file configuration interface and send these parameters to the back-end. The back-end retrieves the corresponding table template from the database based on the front-end configuration parameters. For example, if the extracted table data has three columns, a three-column table template is used; if it has six columns, a six-column table template is used. The extracted table data is then converted into standard table data based on the template. For instance, if only two columns of table data are needed, the data in the template is converted into two-column standard table data. Through these processing steps, various types of volume tables are converted into editable and controllable standard table data in a unified format.

[0025] Step S5: In response to receiving a data verification request sent by the user through the front end, the back end calls the corresponding data verification model in the database according to the configuration parameters of the front end to verify the standard table data and obtain the target table data. Please see Figure 3 , Figure 3 The flowchart for data verification provided in the embodiments of this application is as follows: Figure 3As shown, when a user needs to verify data, a data verification request is sent through the frontend. The backend reads standard table data, obtains the frontend's configuration parameters, and retrieves the corresponding data verification model from the database based on these parameters. The backend uses the data verification model to evaluate whether there are any outliers in the standard table data obtained in the previous steps. If no outliers are found, the standard table data is stored in the database as the target table data. If outliers are found, the data verification result is displayed on the frontend through a secondary confirmation interface, highlighting the outliers to facilitate secondary confirmation and data correction by the user.

[0026] Step S7: In response to receiving a data storage request sent by the user through the front end, the back end stores the target table data in the database.

[0027] When users need to store data, the frontend sends a data storage request, and the backend stores the target table data into the corresponding database location based on the frontend's configuration parameters. Data can be stored as an Excel document or a MySQL database. The former utilizes Python libraries such as pandas, openpyxl, and xlrd for easy data export to users, while the latter uses a Django framework with a dedicated database model for persistent system storage.

[0028] The volume table integrated management method provided in this application embodiment is applied to a volume table integrated management system, which includes a front-end, a back-end, and a database. The method includes: in response to receiving a data extraction request sent by a user through the front-end, the back-end executes the corresponding data extraction program according to the source file type of the volume table to obtain extracted table data; in response to receiving a data processing request sent by a user through the front-end, the back-end calls the corresponding table template in the database according to the configuration parameters of the front-end, and converts the extracted table data into standard table data according to the table template; in response to receiving a data verification request sent by a user through the front-end, the back-end calls the corresponding data verification model in the database according to the configuration parameters of the front-end to verify the standard table data to obtain target table data; in response to receiving a data storage request sent by a user through the front-end, the back-end stores the target table data in the database. This solves the technical problem of poor data management effect of volume tables in related technologies and achieves the technical effect of accurate, efficient, and unified volume table data management.

[0029] In some embodiments, the volume table integrated management method further includes: in response to receiving a login request sent by a user through the front end, the login request including login information, the back end verifies the login information based on user data in the database, and if the verification is successful, sends the corresponding token to the front end.

[0030] When a user needs to log in, a login request is sent through the front-end user login interface. The back-end verifies the login information in the request. If the verification is successful, a corresponding token is sent to the front-end. Specifically, the front-end sends the login information, including the username and password, to the back-end via an HTTP POST request. The back-end matches the login information with the user data in the database to determine if the user exists. If the user exists, a corresponding token is generated and sent to the front-end. Subsequent operations by the user, such as loading, deleting, retrieving, extracting, and validating files, require this token.

[0031] In some embodiments, the volume table integrated management method further includes: in response to receiving a data query request sent by a user through the front end, the back end encapsulates the corresponding data in the database into JSON format and sends it to the front end via WebSocket.

[0032] When a user needs to query data, the frontend sends a first data query request to the backend. Based on this request, the backend encapsulates the corresponding data from the database into JSON format and sends it to the frontend's data storage and query interface via WebSocket for display. The JSON format includes a data packet header, timestamp, data packet number, body content, and transmission verification.

[0033] In some embodiments, the volume table integrated management system further includes an external interface, and the above-mentioned volume table integrated management method further includes: in response to receiving a data query request sent by a user through the external interface, the backend encapsulates the corresponding data in the database into JSON format and sends it to the external interface via WebSocket.

[0034] When an external system needs to query data, it sends a second data query request to the backend through the external interface. The external interface and the backend are connected via WebSocket. The backend authenticates the interface token in the second data query request, determining its permission scope. If the user lacks query permissions, the query fails; otherwise, the backend encapsulates the queried data into JSON format and sends it to the external interface via WebSocket.

[0035] In some embodiments, the source file types of the volume table include native PDF, scanned PDF, and Excel. The backend executes the corresponding data extraction program according to the source file type of the volume table to obtain the extracted tabular data. Specifically, this includes: if the source file type of the volume table is native PDF, then the data is extracted using the Python tabula-py library; if the source file type of the volume table is scanned PDF, then the scanned PDF is converted into an image, and data in the image is extracted and recognized using OCR; if the source file type of the volume table is Excel format, then the data in the Excel document is read using the Python pandas, openpyxl, and xlrd libraries.

[0036] Please see Figure 4 , Figure 4 A flowchart of the volume table integrated management method provided in the embodiments of this application is shown below. Figure 4 As shown, for native PDFs, professional PDF processing software such as Adobe Acrobat or Foxit PhantomPDF is used to filter background images and remove watermarks, retaining only the text data. Based on the front-end configuration parameters, the back-end Python program selects the corresponding processing function and table template to complete the conversion from native PDF to standard table data. For scanned PDFs, the PDF document is first converted into an image array. Then, each image in the array is filtered to remove invalid information such as watermarks, stamps, and noise. OCR technology is then used to extract valid text content from the images, resulting in extracted table data. The corresponding table template is obtained based on the front-end configuration parameters, and the extracted table data is converted into standard table data according to the template. For Excel documents, standard table data is directly extracted. Different source file types for different volume tables correspond to different data extraction programs, realizing the conversion of source file types of different volume tables into unified table data, facilitating subsequent data processing. Here, OCR refers to optical character recognition technology.

[0037] In some embodiments, as shown in the figure, the extraction and recognition of data in an image using OCR specifically includes: preprocessing the image, including excluding non-target elements, performing grayscale conversion, binarization, and morphological processing on the image, and correcting the image using affine transformation; based on the preprocessed image, summing the non-zero pixels in each row of the image, establishing a cumulative frequency histogram, finding and adding appropriate row dividing lines based on the cumulative frequency histogram, and obtaining an image divided into rows; based on the image divided into rows, extracting and recognizing the data content and position information of each text block in the image using OCR; generating a first dot matrix image based on the position information of each text block; clustering the first dot matrix image according to the size of the horizontal and vertical coordinates to generate a second dot matrix image; and obtaining the extracted tabular data according to the mapping relationship between the first and second dot matrix images.

[0038] The above OCR extraction and recognition methods enable accurate identification and extraction of volume table data from scanned PDFs, which facilitates accurate and efficient data management of volume tables.

[0039] In some embodiments, standard table data is validated to obtain target table data. Specifically, this includes: using a data validation model to perform anomaly detection on the standard table data, including jump value detection, missing value detection, duplicate paragraph detection, and human tampering trace detection, to obtain anomaly detection results.

[0040] The frontend sends a data validation request via POST, with configuration parameters set by the frontend, including the user-selected spreadsheet file and cabin number. The backend then calls the corresponding data validation model from the database based on these parameters. The data validation model performs anomaly detection, including jump value detection, missing value detection, duplicate paragraph detection, and human tampering detection, obtaining the corresponding anomaly detection results (jump values, missing values, duplicate values, etc.). The backend then sends these results to the frontend for visualization.

[0041] In some embodiments, validating standard table data to obtain target table data further includes: the backend sending anomaly detection results to the frontend, and the user correcting the standard table data according to the anomaly detection results to obtain the target table data.

[0042] Users correct errors in the corresponding spreadsheet files based on anomaly detection results. Users send storage requests from the front-end, and the back-end stores the target spreadsheet data in the database.

[0043] In some embodiments, the front-end sets configuration parameters, the user clicks save, the front-end passes the configuration parameters to the back-end via a POST request, and the back-end places the configuration parameters in the configuration folder set by the user.

[0044] In some embodiments, when a user clicks on the data preview, the front end will initiate a POST request. The back end will first read the corresponding configuration parameters, read the corresponding table file, save it as an image in the corresponding configuration folder, and then read the image, read the volume data in the image, and send it to the front end for display.

[0045] In some embodiments, when a user clicks to get or delete a file, the front end sends a POST request to the back end; when a user clicks to download a file, the front end sends a GET request to the back end. The back end executes the corresponding operation according to the request instructions. The back end traverses the corresponding user files, obtains the file name and size information, and then sends the data to the front end by compressing the data into a zip package.

[0046] This application provides a volume table integrated management system, which includes a front-end, a back-end, and a database. The front-end is used to send data extraction requests, data processing requests, data verification requests, and data storage requests to the back-end; it is also used to configure parameters and send them to the back-end. The back-end is used to respond to a user's data extraction request sent through the front-end by executing the corresponding data extraction program based on the source file type of the volume table to obtain extracted table data; to respond to a user's data processing request sent through the front-end by calling the corresponding table template in the database according to the front-end's configuration parameters, and converting the extracted table data into standard table data based on the table template; to respond to a user's data verification request sent through the front-end by calling the corresponding data verification model in the database according to the front-end's configuration parameters to verify the standard table data and obtain target table data; and to respond to a user's data storage request sent through the front-end by storing the target table data in the database. The database is used to store user data, table templates, data verification models, and target table data.

[0047] In some embodiments, the backend is further configured to respond to receiving a login request sent by a user through the frontend, the login request including login information, the backend verifying the login information based on user data in the database, and if the verification is successful, sending the corresponding token to the frontend.

[0048] In some embodiments, the backend is further configured to respond to receiving a first data query request sent by the user through the frontend, encapsulate the corresponding data in the database into JSON format, and send it to the frontend via WebSocket.

[0049] In some embodiments, the volume table integrated management system also includes an external interface. The backend is further configured to respond to receiving a second data query request sent by a user through the external interface, and the backend encapsulates the corresponding data in the database into JSON format and sends it to the external interface via WebSocket.

[0050] In some embodiments, the backend specifically includes: a user management module, configured to respond to receiving a login request sent by a user through the frontend, the login request including login information, verifying the login information according to user data in the database, and if the verification is successful, sending the corresponding token to the frontend; a data extraction module, configured to respond to receiving a data extraction request sent by a user through the frontend, executing the corresponding data extraction program according to the source file type of the volume table to obtain the extracted table data; a data processing module, configured to respond to receiving a data processing request sent by a user through the frontend, calling the corresponding table template in the database according to the frontend configuration parameters, and converting the extracted table data into standard table data according to the table template; and a data verification module, configured to respond to receiving a data verification request sent by a user through the frontend. Based on the verification request, the system calls the corresponding data verification model in the database according to the configuration parameters of the front end to verify the standard table data and obtain the target table data. The data storage and query module is used to respond to the data storage request sent by the user through the front end and store the target table data in the database; and to respond to the first data query request sent by the user through the front end, encapsulate the corresponding data in the database into JSON format and send it to the front end via WebSocket; and to respond to the second data query request sent by the user through the external interface, encapsulate the corresponding data in the database into JSON format and send it to the external interface via WebSocket. The external interface is used to receive the second data query request from the external system.

[0051] Specifically, the backend includes: a user management module, responsible for user login verification, user registration, and granting user tokens; and a data extraction module, used to read the source file types of the volume table, including scanned PDFs, native PDFs, and Excel files. For native PDFs, Adobe Acrobat or Foxit is used. Professional PDF processing software such as PhantomPDF performs background image filtering and watermark removal, retaining only the text data to obtain extracted table data. For scanned PDFs, the PDF document is first converted into an image array, then each image in the array is filtered to remove invalid information such as watermarks, stamps, and noise. Then, OCR technology is used to extract the valid text content from the images to obtain extracted table data. For Excel documents, the extracted table data can be obtained directly. The data processing module is used to standardize the extracted table data into standard table data for easier subsequent processing. This includes calling the corresponding table template in the database according to the front-end configuration parameters and readjusting the format of the extracted table data according to the template to generate standard table data. The data validation module is used to determine whether there are abrupt values, missing values, duplicate paragraphs, or artificial modification traces in the standard table data. The data storage and query module is responsible for user data query and storing the target table data into the database.

[0052] In some embodiments, the front end specifically includes: a user login interface for user registration and login; a file configuration interface for users to configure parameters; a data display interface for displaying data; a secondary confirmation interface for displaying anomaly detection results; a file management interface for users to download, delete, and display existing files in the database; and a data storage and query interface for querying the volume table information stored in the database.

[0053] In some embodiments, please refer to Figure 5 , Figure 5 This is a schematic diagram of the volumetric meter integrated management system provided in an embodiment of this application.

[0054] In some embodiments, the file configuration interface and the data display interface can be set up side by side on the same interface, which can be used for data preview. See also Figure 6 , Figure 6 This is a schematic diagram of a data preview interface provided for an embodiment of this application. Figure 6 As shown, the left side of the interface is the file configuration interface, the middle is a preview image of the source file of the volume table, and the right side is a display interface showing the standard table data obtained after the extraction process. By comparing the preview image of the source file of the volume table and the standard table data, the data extraction effect can be easily and intuitively seen.

[0055] In some embodiments, the volume table integrated management system includes a Vue frontend, a Django backend, and a MySQL database. Nginx is used as a reverse proxy server, providing access points for Vue static resources and Django backend APIs, thus connecting the frontend and backend.

[0056] When a user visits the website, Nginx directly returns the Vue frontend pages and resources (including images, CSS, JS files, etc.) to the user. A packaged Vue project typically generates a `dist` folder containing all static files. Configuring the Nginx folder location facilitates direct loading of these files. To allow the frontend to access backend APIs, a path is configured in Nginx. When a user accesses this path through the Vue frontend, Nginx forwards the request to the Django backend. The Vue frontend and Django backend are usually deployed on the same domain. If they run on different ports or domains, cross-origin issues will occur. Adding cross-origin headers to Nginx can resolve these issues. The Django backend has built-in interfaces for accessing the database. During runtime, the Django backend connects to the database based on the frontend's configuration parameters. The Django backend simply calls the corresponding API functions to add or retrieve information from the database. In the Django backend, generating migration files and application migrations are two key steps in database operations, used to reflect model changes in the database. The essence of migration is to convert the data structures (such as fields and tables) in the model into operation statements that the database can understand, thereby synchronizing the database structure with Django's model code.

[0057] In this embodiment, the volume table integrated management system is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0058] Please see Figure 7 , Figure 7 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 7As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0059] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0060] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0061] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0063] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0064] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0065] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of any embodiment of this application.

[0066] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

[0067] The systems or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0068] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0074] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0075] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0076] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for comprehensive management of volumetric tables, characterized in that, The method is applied to a volume table integrated management system, which includes a front-end, a back-end, and a database. The method includes: In response to receiving a data extraction request sent by the user through the front end, the back end executes the corresponding data extraction program according to the source file type of the volume table to obtain the extracted table data; In response to receiving a data processing request sent by the user through the front end, the back end calls the corresponding table template in the database according to the front end's configuration parameters, and converts the extracted table data into standard table data according to the table template. In response to receiving a data verification request sent by the user through the front end, the back end calls the corresponding data verification model in the database according to the front end's configuration parameters to verify the standard table data and obtain the target table data. In response to receiving a data storage request from the user via the front end, the back end stores the target table data in the database.

2. The method according to claim 1, characterized in that, The method further includes: In response to receiving a login request sent by a user through the front end, the back end verifies the login information based on the user data in the database. If the verification is successful, the back end sends the corresponding token to the front end.

3. The method according to claim 1, characterized in that, The method further includes: In response to receiving the first data query request sent by the user through the front end, the back end encapsulates the corresponding data in the database into JSON format and sends it to the front end via WebSocket.

4. The method according to claim 1, characterized in that, The volumetric meter integrated management system also includes an external interface, and the method further includes: In response to receiving a second data query request from a user through an external interface, the backend encapsulates the corresponding data in the database into JSON format and sends it to the external interface via WebSocket.

5. The method according to claim 1, characterized in that, The source file types for the volume table include native PDF, scanned PDF, and Excel. The backend executes the corresponding data extraction program based on the source file type of the volume table to obtain the extracted tabular data, specifically including: If the source file type of the volume table is a native PDF, then the data is extracted using Python's tabula-py library; If the source file type of the volume table is a scanned PDF, then the scanned PDF is converted into an image, and OCR is used to extract and recognize the data in the image; If the source file of the volume table is in Excel format, then use Python's pandas, openpyxl, and xlrd libraries to read the data from the Excel document.

6. The method according to claim 5, characterized in that, The method of extracting and recognizing data from images using OCR specifically includes: Image preprocessing includes removing non-target elements from the image, performing grayscale conversion, binarization, and morphological processing on the image, and correcting the image using affine transformation; Based on the preprocessed image, the sum of non-zero pixels in each row of the image is counted, a cumulative frequency histogram is built, and based on the cumulative frequency histogram, appropriate row dividing lines are found and added to obtain an image divided into blocks by row. Based on the image segmented by line, OCR is used to extract and recognize the data content and position information of each text block in the image. Based on the position information of each text block, generate the first bitmap; The first dot matrix image is clustered according to the magnitude of the horizontal and vertical axes to generate the second dot matrix image. Based on the mapping relationship between the first dot matrix and the second dot matrix, the extracted tabular data is obtained.

7. The method according to claim 1, characterized in that, The validation of the standard table data specifically includes: Anomaly detection is performed on standard table data using a data validation model, including anomaly detection, missing value detection, duplicate paragraph detection, and detection of traces of human tampering, to obtain anomaly detection results.

8. The method according to claim 7, characterized in that, The method further includes: The backend sends the anomaly detection results to the frontend, and the user corrects the standard table data based on the anomaly detection results to obtain the target table data.

9. A comprehensive management system for volumetric meters, characterized in that, The volume table integrated management system includes a front-end, a back-end, and a database, wherein: The frontend is used to send data extraction requests, data processing requests, data validation requests, and data storage requests to the backend; it is also used to configure parameters and send them to the backend. The backend is used to respond to data extraction requests sent by users through the frontend, execute the corresponding data extraction program according to the source file type of the volume table, and obtain the extracted table data; to respond to data processing requests sent by users through the frontend, call the corresponding table template in the database according to the frontend configuration parameters, and convert the extracted table data into standard table data according to the table template; to respond to data verification requests sent by users through the frontend, call the corresponding data verification model in the database according to the frontend configuration parameters, verify the standard table data, and obtain the target table data; and to respond to data storage requests sent by users through the frontend, store the target table data in the database. A database is used to store table templates, data validation models, and target table data.

10. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the volume table integrated management method according to any one of claims 1 to 8 by executing the computer instructions.

Images