FastAPI-based JSON data processing system implementation method

Abstract

The application discloses a JSON data processing system implementation method based on FastAPI, which comprises the following steps: establishing a working directory based on FastAPI, creating a request processor in the working directory, and receiving original JSON data through the request processor; constructing a database object model in the working directory, generating a database table, and storing the original JSON data into the database table; constructing a business verification model in the working directory, preprocessing the original JSON data through the business verification model, and converting the processed JSON data into a Python object; creating a business processing tool in the working directory, processing the business logic of the Python object through the business processing tool, converting the processed Python object into a JSON data format; and creating a request agent and a middleware client agent of an external system in the working directory, and interacting the converted JSON data with the external system and the middleware. The application realizes an asynchronous processing kernel, supports high-concurrency tasks, improves the execution efficiency, and reduces the response time.

Description

Technical Field

[0001] This invention belongs to the field of web backend applications and discloses a method for implementing a JSON data processing system based on FastAPI. Background Technology

[0002] With the rapid development of the Internet and the maturity of big data technology, more and more application scenarios need to process and analyze large amounts of JSON data. Existing application systems can often only process the kernel synchronously, which cannot support high-concurrency tasks and has a slow response speed. Summary of the Invention

[0003] Purpose of the invention: To provide a method for implementing a JSON data processing system based on FastAPI, so as to solve the above-mentioned problems existing in the prior art.

[0004] Technical solution: A method for implementing a JSON data processing system based on FastAPI, including the following steps:

[0005] S1. Establish a working directory based on FastAPI, create a request handler in the working directory, and receive raw JSON data through the request handler;

[0006] S2. Build a database object model in the working directory, generate database tables, and store the raw JSON data in the database tables;

[0007] S3. Build a business validation model in the working directory, preprocess the original JSON data through the business validation model, and convert the processed JSON data into Python objects.

[0008] S4. Create a business processing tool in the working directory, use the business processing tool to process the business logic of Python objects, convert the processed Python objects into JSON data format, and store them in the database table.

[0009] S5. Create a request proxy for the external system and a middleware client proxy in the working directory to interact with the converted JSON data with the external system and middleware to achieve asynchronous processing.

[0010] According to one aspect of this application, step S1 specifically comprises:

[0011] S11. Install the FastAPI library on the terminal, create a working directory based on the FastAPI library, and create a routing packet in the working directory;

[0012] S12. Create a request handler in the routing packet and receive raw JSON data through the request handler;

[0013] S13. Define the format and type of raw JSON data based on the FastAPI library.

[0014] According to one aspect of this application, step S2 specifically comprises:

[0015] S21. Create a data package in the working directory and import the Tortoise-ORM library and MySQL library into the data package;

[0016] S22. Based on the Tortoise-ORM library and MySQL library, construct a database object model in the data packet, generate a database table, and store the raw JSON data in the database table;

[0017] S23. Bind the database object model and request handler to establish a mapping relationship between the raw JSON data and the database table.

[0018] According to one aspect of this application, step S3 specifically comprises:

[0019] S31. Create a controller package in the working directory and import the pydantic library into the controller package;

[0020] S32. Based on the pydantic library, build a business validation model in the controller package, perform data validation and formatting on the original JSON data, and convert the processed JSON data into Python objects.

[0021] S33. Bind the business validation model and the request handler to establish a mapping relationship between the raw JSON data, the business validation model, the Python object, and the database table.

[0022] According to one aspect of this application, step S4 specifically comprises:

[0023] S41. Create a controller package in the working directory, import the pydantic library into the controller package, and create a business processing tool based on the pydantic library in the controller package;

[0024] S42. Based on business processing tools, perform business logic processing on Python objects, including calculation, analysis, filtering, sorting, and aggregation;

[0025] S43. Convert the processed Python object into JSON data and store the converted JSON data in a database table;

[0026] S44. Bind the business processing tools and request handlers to establish a mapping relationship between the original JSON data, business validation model, Python object, business processing tools, transformed JSON data, and database table.

[0027] According to one aspect of this application, step S5 specifically comprises:

[0028] S51. Create a dependency package in the working directory and import the Python library into the dependency package;

[0029] S52. Based on Python libraries, create request proxies and middleware client proxies for external systems in the dependency packages;

[0030] S53. In the request broker of the external system, create at least one request broker function, using the aio library, to asynchronously send and receive the converted JSON data;

[0031] S54. In the middleware client proxy, create at least one middleware client proxy function that, based on a Python library, asynchronously publishes and subscribes to the transformed JSON data.

[0032] According to one aspect of this application, step S12 specifically comprises:

[0033] S12a. Create a request handler in the routing packet, import the cryptography library in the request handler, use the Fernet class of the cryptography library to generate a symmetric encryption key, and obtain the key object.

[0034] S12b: Receive the raw JSON data through the request processor, encrypt the raw JSON data based on the key object, and obtain the encrypted binary data;

[0035] S12c: Encode the encrypted binary data to obtain the encoded string data;

[0036] S12d converts the encoded string data into JSON format data to obtain encrypted JSON data.

[0037] According to one aspect of this application, step S12 may also be:

[0038] S121. Create a request handler in the routing packet and import the zlib library into the request handler;

[0039] S122. Receive the raw JSON data through the request processor, use the compress function of the zlib library to compress the raw JSON data, and obtain the compressed binary data.

[0040] S123. Encode the compressed binary data to obtain encoded string data;

[0041] S124. Convert the encoded string data into JSON format data to obtain compressed JSON data.

[0042] According to one aspect of this application, step S42 specifically comprises:

[0043] S42a. Import the scikit-learn library into the business processing tool, and create data mining and analysis functions based on the scikit-learn library;

[0044] S42b: Call the data mining function to perform data mining on the Python object, and call the data analysis function to perform data analysis on the data-mined Python object;

[0045] S42c is a business processing tool that performs business logic processing on Python objects after data analysis, including calculation, analysis, filtering, sorting, and aggregation.

[0046] According to one aspect of this application, it also includes

[0047] S6. Create a visualization tool in the working directory to visualize the converted JSON data and generate charts and reports.

[0048] Beneficial effects: This invention implements an asynchronous processing kernel through a JSON data processing system based on FastAPI, supports high-concurrency tasks, improves execution efficiency, reduces response time, and is suitable for processing and analyzing various large-scale data and scenarios. Attached Figure Description

[0049] Figure 1 This is a flowchart of the present invention.

[0050] Figure 2 This is a flowchart of step S1 of the present invention.

[0051] Figure 3 This is a flowchart of step S2 of the present invention.

[0052] Figure 4 This is a flowchart of step S3 of the present invention.

[0053] Figure 5 This is a flowchart of step S4 of the present invention.

[0054] Figure 6 This is a flowchart of step S5 of the present invention.

[0055] Figure 7 This is a data flow diagram of the present invention.

[0056] Figure 8 This is a design diagram of the processing procedure of the present invention. Detailed Implementation

[0057] like Figure 1 As shown, this invention proposes a method for implementing a JSON data processing system based on FastAPI. The algorithm includes the following steps:

[0058] S1. Establish a working directory based on FastAPI, create a request handler in the working directory, and receive raw JSON data through the request handler;

[0059] S2. Build a database object model in the working directory, generate database tables, and store the raw JSON data in the database tables;

[0060] S3. Build a business validation model in the working directory, preprocess the original JSON data through the business validation model, and convert the processed JSON data into Python objects.

[0061] S4. Create a business processing tool in the working directory, use the business processing tool to process the business logic of Python objects, convert the processed Python objects into JSON data format, and store them in the database table.

[0062] S5. Create a request proxy for the external system and a middleware client proxy in the working directory to interact with the converted JSON data with the external system and middleware to achieve asynchronous processing.

[0063] In a further embodiment, firstly, a working directory is established based on FastAPI, a high-performance asynchronous web framework that enables rapid development and deployment of web applications. Functions for receiving and processing HTTP requests are created in the working directory, using FastAPI's route decorators to define the request paths and methods to receive raw JSON data. JSON data is a lightweight data exchange format that can be easily transmitted between different systems and languages.

[0064] Secondly, a database object model is built in the working directory. A database object model is a class used to describe the structure and relationships of database tables. This can be implemented using the Tortoise-ORM library, an asynchronous ORM framework based on Python that supports various database backends, such as MySQL. The Tortoise-ORM library's `init` function is used to automatically create the database tables, which are two-dimensional structures used to store data. The Tortoise-ORM library's `save` function is then used to store the raw JSON data into the database tables.

[0065] Next, the pydantic library is used to build a business validation model in the working directory. This model is a class used to validate and format raw JSON data. The pydantic library is a Python-based data validation and settings management library that supports various data types and annotations. The `parse_obj_as` function from the pydantic library is used to preprocess the raw JSON data through the business validation model, converting the data into a format and type that meets business requirements. Finally, the `from_orm` function from the pydantic library is used to convert the processed JSON data into Python objects, transforming the data into an in-memory structure that is easy to manipulate and process.

[0066] Next, the pydantic library is used to create a business processing tool in the working directory. This tool is a function used to perform business logic processing on Python objects. It allows for calculation, analysis, filtering, sorting, and aggregation of data based on business requirements, using built-in Python functions or third-party libraries such as scikit-learn. The pydantic library's `json` function is then used to convert the processed Python objects into JSON data format, transforming the data into a text structure that is easy to transmit and interact with. Finally, the Tortoise-ORM library's `save` function is used to store the data in a database table.

[0067] Finally, use Python's standard library or a third-party library to create a request proxy and middleware client proxy for the external system in the working directory. The request proxy and middleware client proxy are two functions used to interact with the external system and middleware, and to interact with the transformed JSON data with the external system and middleware in an asynchronous manner to improve efficiency and performance.

[0068] This embodiment achieves asynchronous processing by preprocessing JSON data, performing business logic processing, and finally interacting with external systems and middleware, thereby improving response speed and concurrency capabilities.

[0069] like Figure 2 As shown, step S1 specifically involves:

[0070] S11. Install the FastAPI library on the terminal, create a working directory based on the FastAPI library, and create a routing packet in the working directory;

[0071] S12. Create a request handler in the routing packet and receive raw JSON data through the request handler;

[0072] S13. Define the format and type of raw JSON data based on the FastAPI library.

[0073] In a further embodiment, the following steps are taken: Install the FastAPI library and all optional dependencies in the terminal; create a working directory named fastapi-demo in the terminal; navigate to the working directory in the terminal; create a routing package named routes in the terminal; create a file named main.py in the routing package to store the main request handler; import the FastAPI library and routing package in the main.py file; create a FastAPI instance named app; define a POST request handler named / data in the main.py file using the @app.post(" / data") decorator; define a parameter named data in the / data request handler, using the Body(...) function to specify that this parameter is JSON data obtained from the request body; return the data parameter as a response in the / data request handler; create a file named models.py in the working directory to store the data model; import the FastAPI library and Pydantic library in the models.py file; create a class named Data that inherits from the BaseModel class; define the properties and types of the raw JSON data in the Data class. Import the models.py file into the main.py file, modify the data parameter of the / data request handler, and specify the type of the parameter using the Data class.

[0074] This embodiment involves installing and initializing the FastAPI framework, creating a web application and API interface, defining and implementing the functionality of the API interface to receive raw JSON data sent by the client, and finally formatting and typering the raw JSON data for subsequent data processing and validation.

[0075] like Figure 3 As shown, step S2 specifically involves:

[0076] S21. Create a data package in the working directory and import the Tortoise-ORM library and MySQL library into the data package;

[0077] S22. Based on the Tortoise-ORM library and MySQL library, construct a database object model in the data packet, generate a database table, and store the raw JSON data in the database table;

[0078] S23. Bind the database object model and request handler to establish a mapping relationship between the raw JSON data and the database table.

[0079] First, create a data package in the working directory, install and import the Tortoise-ORM library and the MySQL library, then define and create database tables, insert raw JSON data into the database tables, and finally achieve bidirectional synchronization between the raw JSON data and the database tables for subsequent data queries and updates.

[0080] In a further embodiment, the Tortoise-ORM library and MySQL library are installed in the terminal; a data package named "data" is created in the working directory; a file named "__init__.py" is created in the data package to initialize the data package, and the Tortoise-ORM library and MySQL library are imported into the "__init__.py" file; a file named "models.py" is created in the data package to store the database object models; the Tortoise-ORM library and the Data class are imported into the models.py file, and a class named "DataModel" is created, which inherits from the Tortoise.Model class; the fields and types of the database tables are defined in the DataModel class; an asynchronous function named "init_db" is defined in the models.py file to initialize the database and generate database tables; the Tortoise.init method is used in the init_db function, passing in the database configuration information; the Tortoise.generate_schemas method is used in the init_db function to generate database tables; the data.models file and the run_async function are imported in the main.py file, the functionality of the / data request handler is modified, the data parameter is used as an instance of the DataModel class, and the save method is used to store it in the database table;

[0081] In the main.py file, import the Tortoise class and use the Tortoise.init_models method, passing in the database configuration information; in the main.py file, use the Tortoise.register_tortoise method, passing in the FastAPI instance app and the database configuration information; in the main.py file, use the app.add_middleware method to add a middleware to manage database connections and shutdowns.

[0082] This embodiment stores the raw JSON data in a database table and facilitates data processing by binding request handlers and database object models, which helps improve data processing efficiency and reliability, and ensures data integrity and consistency.

[0083] like Figure 4 As shown, step S3 specifically involves:

[0084] S31. Create a controller package in the working directory and import the pydantic library into the controller package;

[0085] S32. Based on the pydantic library, build a business validation model in the controller package, perform data validation and formatting on the original JSON data, and convert the processed JSON data into Python objects.

[0086] S33. Bind the business validation model and the request handler to establish a mapping relationship between the raw JSON data, the business validation model, the Python object, and the database table.

[0087] First, create a controller package in the working directory. Then, install and import the pydantic library into the controller package to use the data validation and transformation functions of the pydantic library. Next, define and implement the business validation model to perform data validation and formatting on the raw JSON data and convert the processed JSON data into Python objects. Finally, bind the business validation model and the request handler to achieve bidirectional synchronization between the raw JSON data, the business validation model, the Python objects, and the database tables for subsequent data queries and updates.

[0088] In a further embodiment, the pydantic library is installed in the terminal; a controller package named `controllers` is created in the working directory; a file named `__init__.py` is created in the controller package to initialize the controller package; the pydantic library is imported in the `__init__.py` file, such as `from pydantic import BaseModel, Field`. In the `models.py` file within the controller package, create a file named `models.py` to store the business validation models. In `models.py`, import the `pydantic` library and the `Data` class, and create a class named `DataValidator` that inherits from `BaseModel`. Define the attributes and types of the raw JSON data in the `DataValidator` class. Define some custom validators in the `DataValidator` class, using the `@validator` decorator to perform business logic validation and processing on the attributes of the raw JSON data, such as checking if `name` is empty and if `age` is within a reasonable range. Define a function named `validate_data` in `models.py` to receive the raw JSON data, create an instance of the `DataValidator` class, call its `validate` method, and return an instance of the `Data` class. In the `main.py` file, import the `controllers.models` file, modify the functionality of the ` / data` request handler, treat the `data` parameter as an instance of the `DataValidator` class, use the `validate_data` function to convert it into an instance of the `Data` class, and use the `save` method to store it in a database table.

[0089] This embodiment uses the Pydantic library for data validation and mapping to ensure data integrity and consistency, while simplifying the data processing flow and improving development efficiency and code readability.

[0090] like Figure 5 As shown, step S4 specifically involves:

[0091] S41. Create a controller package in the working directory, import the pydantic library into the controller package, and create a business processing tool based on the pydantic library in the controller package;

[0092] S42. Based on business processing tools, perform business logic processing on Python objects, including calculation, analysis, filtering, sorting, and aggregation;

[0093] S43. Convert the processed Python object into JSON data and store the converted JSON data in a database table;

[0094] S44. Bind the business processing tools and request handlers to establish a mapping relationship between the original JSON data, business validation model, Python object, business processing tools, transformed JSON data, and database table.

[0095] Create a controller package in the working directory, install and import the pydantic library into the controller package to utilize pydantic's data transformation and serialization functions. Create a business processing tool in the controller package to perform business logic processing on Python objects. Then define and implement the functionality of the business processing tool to perform business logic processing on Python objects, including calculation, analysis, filtering, sorting, and aggregation. Convert the processed Python objects into JSON data for interaction with external systems and middleware. Store the converted JSON data in a database table for subsequent data querying and updates. Finally, bind the business processing tool and request handler to achieve bidirectional synchronization of the original JSON data, business validation model, Python objects, business processing tool, converted JSON data, and database table for subsequent data querying and updates.

[0096] In a further embodiment, the pydantic library is installed in the terminal; a controller package named `controllers` is created in the working directory; a file named `__init__.py` is created in the controller package to initialize the controller package; the pydantic library is imported into the `__init__.py` file; a file named `tools.py` is created in the controller package to store business processing tools; the pydantic library and the `Data` class are imported into the `tools.py` file, and a class named `DataProcessor`, inheriting from the `BaseModel` class, is created; the attributes and types of Python objects are defined in the `DataProcessor` class; some custom processors are defined in the `DataProcessor` class, using the `@validator` decorator to perform some business logic processing on the attributes of Python objects; a function named `process_data` is defined in the `tools.py` file to receive an instance of the `Data` class, create an instance of the `DataProcessor` class, call its `process` method, and return an instance of the `Data` class; the `json` library is imported into the `tools.py` file, and the functionality of the `process_data` function is modified to convert the returned instance of the `Data` class into JSON data using the `json.dumps` method;

[0097] In the main.py file, import the data.models file and the run_async function. Modify the functionality of the / data request handler to treat the data parameter as an instance of the Data class, use the process_data function to convert it into JSON data, and use the save method to store it in the database table. In the main.py file, import the controllers.tools file and modify the functionality of the / data request handler to treat the data parameter as an instance of the DataValidator class, use the validate_data function to convert it into an instance of the Data class, use the process_data function to convert it into JSON data, and use the save method to store it in the database table.

[0098] This embodiment uses the Pydantic library and business processing tools to process and transform data more efficiently, while simplifying the data processing flow, improving development efficiency and code readability. In addition, by establishing mapping relationships, it can better understand and track the flow and processing of data.

[0099] like Figure 6 As shown, step S5 specifically involves:

[0100] S51. Create a dependency package in the working directory and import the Python library into the dependency package;

[0101] S52. Based on Python libraries, create request proxies and middleware client proxies for external systems in the dependency packages;

[0102] S53. In the request broker of the external system, create at least one request broker function, using the aio library, to asynchronously send and receive the converted JSON data;

[0103] S54. In the middleware client proxy, create at least one middleware client proxy function that, based on a Python library, asynchronously publishes and subscribes to the transformed JSON data.

[0104] Create dependency packages in the working directory, install and import Python libraries within these packages to utilize their asynchronous programming and network communication capabilities. Define and create request brokers and middleware client brokers for external systems within these packages to interact with external systems and middleware, sending and receiving transformed JSON data. Implement one or more request broker functions in the external system's request broker to asynchronously send and receive transformed JSON data. Implement one or more middleware client broker functions in the middleware client broker to asynchronously publish and subscribe to transformed JSON data.

[0105] In a further embodiment, the aiohttp and pika libraries are installed in the terminal; a dependency package named dependencies is created in the working directory; a file named __init__.py is created in the dependency package to initialize the dependency package, and Python libraries are imported in the __init__.py file; a file named proxies.py is created in the dependency package to store request proxies and middleware client proxies for external systems.

[0106] In the `proxies.py` file, import the Python library. Create a class named `ExternalSystemProxy` to encapsulate the request proxy functionality of the external system. Define a method named `__init__` within the `ExternalSystemProxy` class to initialize the URL and HTTP session of the external system. Define an asynchronous method named `send_data` within the `ExternalSystemProxy` class to send the transformed JSON data to the external system and return a response. In the `proxies.py` file, create a class named `MiddlewareClientProxy` to encapsulate the middleware client proxy functionality. Define a method named `__init__` within the `MiddlewareClientProxy` class to initialize the middleware's connection parameters and channels. Define an asynchronous method named `publish_data` within the `MiddlewareClientProxy` class to publish the transformed JSON data to the middleware's exchange and return an acknowledgment. Define an asynchronous method named `subscribe_data` within the `MiddlewareClientProxy` class to subscribe to the middleware's queue and return the received JSON data.

[0107] In the `proxies.py` file, import the `aio` library, e.g., `import aiofiles`. Define an asynchronous function named `send_to_system_a` in `proxies.py` to send converted JSON data to system A and return a response. In the `send_to_system_a` function, create an instance of the `ExternalSystemProxy` class, passing in the URL of system A. In the `send_to_system_a` function, use the `aiofiles` library to asynchronously open a file named `data.json` and read the JSON data from it. In the `send_to_system_a` function, use the `send_data` method of `system_a_proxy` to asynchronously send JSON data to system A and return a response. Finally, return the response within the `send_to_system_a` function.

[0108] In the `proxies.py` file, import Python libraries such as `json`. Define an asynchronous function named `publish_to_middleware` to publish the transformed JSON data to the middleware and return an acknowledgment. Within the `publish_to_middleware` function, create an instance of the `MiddlewareClientProxy` class, passing in the middleware's connection parameters. Use the `json` library to encode the transformed JSON data into a byte string within the `publish_to_middleware` function. Use the `publish_data` method of `middleware_proxy` to asynchronously publish the JSON data to the middleware's exchange and return an acknowledgment. Return an acknowledgment within the `publish_to_middleware` function. Finally, in `proxies.py`... The `y` file defines an asynchronous function named `subscribe_from_middleware` to subscribe to the middleware's queue and return the received JSON data. Within the `subscribe_from_middleware` function, an instance of the `MiddlewareClientProxy` class is created, passing in the middleware's connection parameters. The `subscribe_data` method of `middleware_proxy` is used within the `subscribe_from_middleware` function to asynchronously subscribe to the middleware's queue and return the received JSON data. The `subscribe_from_middleware` function also uses a json library to decode the received byte string into JSON data. Finally, the `subscribe_from_middleware` function returns the JSON data.

[0109] In a further embodiment,

[0110] When creating a request handler, use FastAPI's dependency injection feature to pass services or components as parameters to achieve better decoupling and reuse.

[0111] When defining the format and type of raw JSON data, you can use FastAPI's request body model and leverage the data validation and type conversion features provided by the Pydantic library to simplify code and improve security.

[0112] When storing raw JSON data into a database table, you can use the asynchronous query functionality provided by the Tortoise-ORM library to improve performance and concurrency.

[0113] When building a business validation model, you can use the custom validators provided by the Pydantic library to implement more complex business logic and error handling.

[0114] When converting processed JSON data into Python objects, you can use the generic model provided by the Pydantic library to achieve more flexible type definitions and conversions.

[0115] When performing business logic processing on Python objects, you can use FastAPI's background task functionality to run some time-consuming tasks in the background, thereby improving response speed and user experience.

[0116] When converting processed Python objects into JSON data, you can use FastAPI's response model and leverage the data serialization and filtering capabilities provided by the Pydantic library to achieve optimized output and security.

[0117] When creating request proxies and middleware client proxies for external systems, you can use FastAPI's HTTP client functionality, leveraging the concise API and efficient connection pooling provided by the requests library to achieve more convenient request sending and receiving.

[0118] When creating visualization tools, you can use FastAPI's template functionality and leverage the powerful template engine provided by the jinja2 library to achieve more aesthetically pleasing page rendering and interaction.

[0119] When visualizing the converted JSON data, you can use FastAPI's static file functionality and leverage the rich charting capabilities provided by the matplotlib library to achieve more diverse data presentation and analysis.

[0120] This embodiment uses request proxy and middleware client proxy to make it easier for the program to interact with other systems, improving the system's scalability and flexibility.

[0121] Furthermore, step S12 specifically involves:

[0122] S12a. Create a request handler in the routing packet, import the cryptography library in the request handler, use the Fernet class of the cryptography library to generate a symmetric encryption key, and obtain the key object.

[0123] S12b: Receive the raw JSON data through the request processor, encrypt the raw JSON data based on the key object, and obtain the encrypted binary data;

[0124] S12c: Encode the encrypted binary data to obtain the encoded string data;

[0125] S12d converts the encoded string data into JSON format data to obtain encrypted JSON data.

[0126] Create a request handler in the routing package, install and import the cryptography library in the request handler to use the symmetric encryption function of the cryptography library, and generate a symmetric encryption key in the request handler to obtain a key object; define and implement the function of the request handler to receive raw JSON data, encrypt the raw JSON data based on the key object to obtain encrypted binary data; encode the encrypted binary data to obtain encoded string data for subsequent data conversion and transmission; convert the encoded string data into JSON format data to obtain encrypted JSON data for interaction with external systems and middleware.

[0127] In a further embodiment, first, type `pip install cryptography` in the terminal to install the cryptography library; create a file named `encrypt.py` in the routing package to store the request handler; import the cryptography and FastAPI libraries in the `encrypt.py` file, such as `from cryptography.fernetimport Fernet` and `from fastapi import FastAPI, Body`; create a FastAPI instance named `app` in the `encrypt.py` file, such as `app = FastAPI()`; use the `generate_key` method of the Fernet class in the `encrypt.py` file to generate a symmetric encryption key and obtain a key object, such as `key = Fernet.generate_key()`.

[0128] Then, define a POST request handler named ` / encrypt` in the `encrypt.py` file, using the `@app.post(" / encrypt")` decorator; define a parameter named `data` in the ` / encrypt` request handler, using the `Body(...)` function to specify that this parameter is JSON data obtained from the request body; use the `encrypt` method of the `Fernet` class in the ` / encrypt` request handler, passing in a key object `key` and JSON data `data`, to encrypt the original JSON data, obtaining encrypted binary data, such as `encrypted_data = Fernet(key).encrypt(data)`; return the encrypted binary data in the ` / encrypt` request handler, such as `return encrypted_data`;

[0129] Next, import the base64 library in the encrypt.py file, such as `import base64`; use the `b64encode` method of the base64 library in the ` / encrypt` request handler to encode the encrypted binary data to obtain the encoded string data, such as `encoded_data = base64.b64encode(encrypted_data)`; return the encoded string data in the ` / encrypt` request handler, such as `return encoded_data`;

[0130] Finally, import the json library in the encrypt.py file, such as import json; use the dumps method of the json library in the / encrypt request handler to convert the encoded string data into JSON format data to obtain encrypted JSON data, such as encrypted_json = json.dumps(encoded_data); return the encrypted JSON data in the / encrypt request handler, such as return encrypted_json.

[0131] This embodiment not only improves data security and processing efficiency, but also facilitates subsequent data use and processing.

[0132] Furthermore, step S12 can also be:

[0133] S121. Create a request handler in the routing packet and import the zlib library into the request handler;

[0134] S122. Receive the raw JSON data through the request processor, use the compress function of the zlib library to compress the raw JSON data, and obtain the compressed binary data.

[0135] S123. Encode the compressed binary data to obtain encoded string data;

[0136] S124. Convert the encoded string data into JSON format data to obtain compressed JSON data.

[0137] In a further embodiment, type `pip install zlib` in the terminal to install the zlib library; create a file named `compress.py` in the routing package to store the request handler; import the zlib library and the FastAPI library in the `compress.py` file, such as `import zlib` and `from fastapi import FastAPI, Body`; create a FastAPI instance in the `compress.py` file named `app`, such as `app = FastAPI()`.

[0138] Define a POST request handler named ` / compress` in the `compress.py` file, using the `@app.post(" / compress")` decorator. In the ` / compress` request handler, define a parameter named `data`, using the `Body(...)` function to specify that this parameter is JSON data retrieved from the request body. In the ` / compress` request handler, use the `compress` function from the zlib library, taking the JSON data `data` as input, to compress the raw JSON data, obtaining compressed binary data, such as `compressed_data = zlib.compress(data)`. Return the compressed binary data in the ` / compress` request handler, such as `return compressed_data`. By defining and implementing the request handler functionality, you can receive raw JSON data through the request handler and use the `compress` function from the zlib library to compress the raw JSON data, obtaining compressed binary data.

[0139] In the `compress.py` file, import the base64 library, such as `import base64`. In the ` / compress` request handler, use the `b64encode` method of the base64 library to encode the compressed binary data, obtaining the encoded string data, such as `encoded_data = base64.b64encode(compressed_data)`. Return the encoded string data in the ` / compress` request handler, such as `return encoded_data`. Encoding the compressed binary data to obtain the encoded string data is necessary for subsequent data conversion and transmission.

[0140] In the `compress.py` file, import the `json` library, such as `import json`. In the ` / compress` request handler, use the `dumps` method of the `json` library to convert encoded string data into JSON format, obtaining compressed JSON data, such as `compressed_json = json.dumps(encoded_data)`. Return the compressed JSON data in the ` / compress` request handler, such as `return compressed_json`. This converts encoded string data into JSON format and compresses it for interaction with external systems and middleware.

[0141] This embodiment reduces data volume by compressing the data, saving storage space and transmission bandwidth, thereby improving data processing efficiency and security, while also facilitating subsequent data use and processing.

[0142] Furthermore, step S42 specifically includes:

[0143] S42a. Import the scikit-learn library into the business processing tool, and create data mining and analysis functions based on the scikit-learn library;

[0144] S42b: Call the data mining function to perform data mining on the Python object, and call the data analysis function to perform data analysis on the data-mined Python object;

[0145] S42c is a business processing tool that performs business logic processing on Python objects after data analysis, including calculation, analysis, filtering, sorting, and aggregation.

[0146] In a further embodiment, the scikit-learn library is installed by typing `pip install scikit-learn` in the terminal; a business processing toolkit named `tools` is created by typing `mkdir tools` in the working directory; a file named `__init__.py` is created in the business processing toolkit to initialize it; the scikit-learn library is imported into the `__init__.py` file; a file named `mining.py` is created in the business processing toolkit to store data mining functions; the scikit-learn library and the `Data` class are imported into the `mining.py` file, and a class named `DataMiner`, inheriting from the `BaseModel` class, is created; the attributes and types of Python objects are defined in the `DataMiner` class; some custom miners are defined in the `DataMiner` class, using the `@validator` decorator to perform data mining processing on the attributes of Python objects, such as using the `cluster.KMeans` method for clustering, the `decomposition.PCA` method for dimensionality reduction, and the `metrics.silhouette_score` method for evaluation; a function named `mine_data` is defined in the `mining.py` file to receive an instance of the `Data` class and create a `DataMiner` class. The code snippet describes a method to call the `mine` method of a class called `Data`, which returns an instance of the `Data` class. It then creates a file named `analysis.py` within the business processing toolkit to store data analysis functions. The `analysis.py` file imports the scikit-learn library and the `Data` class, and creates a class named `DataAnalyzer` that inherits from `BaseModel`. The `DataAnalyzer` class defines the attributes and types of Python objects. It also defines custom analyzers within the `DataAnalyzer` class, using the `@validator` decorator to perform data analysis on the attributes of Python objects, such as calculating the mean squared error using `metrics.mean_squared_error`, calculating the accuracy using `metrics.accuracy_score`, and calculating the confusion matrix using `metrics.confusion_matrix`. Finally, the `analysis.py` file defines a function named `analyze_data` that receives an instance of the `Data` class, creates an instance of the `DataAnalyzer` class, calls its `analyze` method, and returns an instance of the `Data` class.By installing and importing the scikit-learn library, you can utilize its data mining and analysis capabilities, as well as create data mining and analysis functions within business processing tools to perform data mining and analysis on Python objects.

[0147] In the main.py file, import the tools.mining and tools.analysis files. Modify the functionality of the / data request handler, treating the data parameter as an instance of the Data class. Use the mine_data function to convert it into an instance of the Data class, and use the analyze_data function to convert it into an instance of the Data class. This implements data mining and analysis capabilities, enabling data mining and analysis of Python objects to obtain data characteristics and patterns.

[0148] In the `tools.py` file, import the `pydantic` library and the `Data` class. Create a class named `DataProcessor`, which inherits from `BaseModel`. Define the attributes and types of Python objects within the `DataProcessor` class. Define some custom processors in the `DataProcessor` class, using the `@validator` decorator, to perform business logic processing on the Python object attributes, such as calculating the length of the name, analyzing the distribution of age, filtering data where age is less than 18, sorting names alphabetically, and aggregating age by age group. Define a function named `process_data` in the `tools.py` file to receive an instance of the `Data` class, create an instance of the `DataProcessor` class, call its `process` method, and return an instance of the `Data` class. By defining and implementing the business logic processing functions, perform business logic processing on the analyzed Python objects, including calculation, analysis, filtering, sorting, and aggregation.

[0149] Furthermore, it also includes S6, creating visualization tools in the working directory, and using these tools to visualize the converted JSON data, generating charts and reports.

[0150] In a further embodiment, type `mkdir visualization` in the working directory to create a visualization toolkit named `visualization`; create a file named `__init__.py` within the visualization toolkit to initialize it; create a file named `plot.py` within the visualization toolkit to store plotting functions; import the matplotlib and seaborn libraries in the `plot.py` file, such as `import matplotlib.pyplot asplt` and `import seaborn as`. In the `plot.py` file, define a function named `plot_data` that receives an instance of the `Data` class. Based on the data type and characteristics, select an appropriate chart type, such as a bar chart, line chart, scatter plot, or box plot. Use functions from the matplotlib and seaborn libraries to plot the data, save the chart as an image file, and return the path to the image file. In the visualization toolkit, create a file named `report.py` to store the report functions. In `report.py`, define a function named `report_data` that receives an instance of the `Data` class. Use functions from the pandas library to perform statistical analysis on the data, obtaining descriptive statistics such as mean, standard deviation, maximum, minimum, and quantiles.

[0151] This embodiment defines and implements the functionality of visualization tools to visualize the converted JSON data, generate charts and reports, and improve the readability and understandability of the data.

[0152] In a further embodiment, such as Figure 7 and Figure 8 As shown, a method for implementing a JSON data processing system based on FastAPI includes:

[0153] Step 1: Create a classic FastAPI project starting directory and write a temporary HTTP request handler in the corresponding router.py file in the router package to accept JSON data.

[0154] Step 2: Create a `database` package in the working directory, and then create a `models` package within it. Write different specific `models.py` files within the `models` package to build the database ORM model based on the Tortoise library and generate the database table structure.

[0155] Step 3: Bind the processors in the router to the ORM model to obtain the mapping relationship between unprocessed data and database tables.

[0156] Step 4: Create a `controller` package in the working directory, and within that `controller` package, create `pydantics` and `handlers` packages. Within the `pydantics` package, write a business validation model based on the `pydantic` library. This model will be used to format, validate, and preprocess externally input JSON data. Create a separate business validation model for each input and output. Because `pydantic` has strong compatibility with Tortoise, you can directly use the ORM mapping business model provided by `pydantic` when writing database business models.

[0157] Step 5: Bind the processors in the router to the business model in PyDantic to obtain the mapping relationship between unprocessed data, data processing model, data processed by the business model, and database table structure. Because PyDantic has strong compatibility with FastAPI, the processors in the router can add an extra layer of HTTP framework-level data validation to external JSON, used to filter invalid input and directly throw an HTTP request parameter error response.

[0158] Step 6: Write temporary business processing tools in the handlers package to handle the business logic generated after the external input JSON is transformed by the business model.

[0159] Step 7: Bind the processors in the router to the business processing tools in the handlers to obtain the mapping relationship between the unprocessed data, the data processing model, the data processed by the business model, the business processing tools, the database table structure, and the business processing tools.

[0160] Step 8: Create a `clients` package in the working directory. Within the `clients` package, write HTTP request brokers and middleware client brokers for different external systems, including other backend services, Redis, Kafka, RabbitMQ, and EMQX. Asynchronous Python libraries such as aiohttp, aioredis, aiokafka, aio-pika, and asyncio-mqtt can be used.

[0161] Step 9: Based on actual needs, write the complete business logic into the corresponding handlers, including data validation, preprocessing, analysis, processing, and storage functions after JSON data input, and reference it in the router. This step largely determines the performance of the entire system.

[0162] For JSON data with similar structures, an abstract approach can be used. An abstract model base class inherits from the `tortoise.model` class, extracting the business primary key of the ORM model. Based on this sequence, CRUD (Create, Read, Update, Delete) methods supporting batch operations can be written. To reduce the number of database requests, when implementing a single batch method, the corresponding actual database data is first read into the program's memory. The program then compares and processes the actual database data with the data in memory before submitting it to the database all at once. A secondary encapsulation is performed on the abstract class, adding some variable attributes for subclasses to implement, thus completing the binding relationship mentioned in step 7. General methods are written in the abstract model class based on these variable attributes, leaving room for possible refactoring or overriding. Entity subclasses inheriting from this abstract class are written based on the actual data, with special handling for methods that need to be overridden or refactored. A larger processing class is encapsulated based on the abstract model class to collect the subclasses of the abstract model class and call their CRUD methods. The actual business logic is implemented in the processing class. In addition to the HTTP interface processor, processors for data generated by middleware consumers can also be written as needed. This creates a processor that analyzes and processes external JSON input before storing it in a database.

[0163] Step 10: Based on possible changes in requirements or version updates, iterate and repeat steps 2 to 9.

[0164] This embodiment can also be implemented using the Flask framework, Oracle database, ActiceMQ as middleware, and hbmqtt and aiormq as asynchronous libraries.

[0165] The system of this application includes the following functional modules: a data analysis module, used for real-time analysis and processing of JSON data, extracting key information and generating reports; a data storage module, used for storing JSON data in a MySQL database for subsequent querying and analysis; and a message sending and receiving module, used for publishing and subscribing to JSON data messages to achieve real-time data transmission and communication.

[0166] This application uses FastAPI, Tortoise-ORM, and MySQL as the backbone to build the web application service. The service center integrates middleware such as Redis, Kafka, EMQX, and RabbitMQ for data analysis, storage, and message sending and receiving. Through this service center, users can easily process and manage JSON data, achieve real-time data analysis and storage, and publish and subscribe to messages. The service center features high performance, high reliability, and scalability, making it suitable for various large-scale data processing and analysis scenarios, including but not limited to the following: e-commerce platforms, combining e-commerce data analysis modules to perform real-time analysis of user behavior data and provide personalized recommendations and marketing strategies; the logistics industry, combining logistics analysis modules to perform real-time analysis of logistics data and optimize route planning and delivery efficiency; the financial industry, combining financial analysis modules to perform real-time analysis of transaction data and provide risk warnings and fraud detection; and social media, combining social media analysis modules to perform real-time analysis of user-generated content and provide personalized recommendations and social network analysis.

[0167] Furthermore, to evaluate the system's performance, we conducted a series of tests. The test results show that the system can process large amounts of JSON data and perform real-time analysis and storage. Its fully asynchronous processing mode supports high concurrency, improving processing speed by 2-10 times and reducing response time by 50%-90% compared to traditional Django projects. The system also exhibits good fault tolerance and recoverability, handling exceptional situations and ensuring data integrity. Simultaneously, the system supports horizontal scaling, allowing the addition of more nodes and middleware as needed.

[0168] In some embodiments, it also includes:

[0169] S1. Establish a working directory based on FastAPI and Serverless Framework, create a request processor in the working directory, and receive raw JSON data through the request processor;

[0170] S2. Build a database object model in the working directory, generate database tables, and store the raw JSON data in the cloud database.

[0171] S3. Build a business validation model in the working directory, preprocess the original JSON data through the business validation model, and convert the processed JSON data into Python objects.

[0172] S4. Create a business processing tool in the working directory, use the business processing tool to process the business logic of Python objects, convert the processed Python objects into JSON data format, and store them in the cloud database.

[0173] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A FastAPI-based JSON data processing system implementation method, characterized in that, The method comprises the following steps: S1, establishing a working directory based on FastAPI, creating a request processor in the working directory, and receiving original JSON data through the request processor; S2, constructing a database object model in the working directory, generating a database table, and storing the original JSON data into the database table; S3, constructing a business verification model in the working directory, preprocessing the original JSON data through the business verification model, and converting the processed JSON data into a Python object; S4, creating a business processing tool in the working directory, processing the business logic of the Python object through the business processing tool, converting the processed Python object into JSON data format, and storing it into the database table; S5, creating a request proxy of an external system and a middleware client proxy in the working directory, interacting the converted JSON data with the external system and the middleware to realize asynchronous processing; Step S5 is specifically: S51, establishing a dependency package in the working directory, and importing a Python library in the dependency package; S52, creating a request proxy of an external system and a middleware client proxy in the dependency package based on the Python library; S53, creating at least one request proxy function in the request proxy of the external system, using the aio library to asynchronously send and receive the converted JSON data; S54, creating at least one middleware client proxy function in the middleware client proxy, based on the Python library, and asynchronously publishing and subscribing the converted JSON data.

2. The FastAPI based JSON data processing system implementation method according to claim 1, wherein, Step S1 is specifically: S11, installing a FastAPI library on a terminal, establishing a working directory based on the FastAPI library, and establishing a routing package in the working directory; S12, creating a request processor in the routing package, and receiving original JSON data through the request processor; S13, defining the format and type of the original JSON data based on the FastAPI library.

3. The FastAPI based JSON data processing system implementation method of claim 1, wherein, Step S2 is specifically: S21, establishing a data package in the working directory, and importing a Tortoise-ORM library and a MySQL library in the data package; S22, constructing a database object model in the data package based on the Tortoise-ORM library and the MySQL library, generating a database table, and storing the original JSON data into the database table; S23, binding the database object model and the request processor to establish a mapping relationship between the original JSON data and the database table.

4. The FastAPI based JSON data processing system implementation method of claim 1, wherein, Step S3 is specifically: S31, establishing a controller package in the working directory, and importing a pydantic library in the controller package; S32, constructing a business verification model in the controller package based on the pydantic library, performing data verification and format processing on the original JSON data, and converting the processed JSON data into a Python object; S33, binding the business verification model and the request processor to establish a mapping relationship among the original JSON data, the business verification model, the Python object, and the database table.

5. The FastAPI based JSON data processing system implementation method of claim 1, wherein, Step S4 is specifically: S41, a controller package is established in the working directory, a pydantic library is imported in the controller package, and a business processing tool is created in the controller package based on the pydantic library; S42, based on the business processing tool, the business logic of the Python object is processed, including calculation, analysis, filtering, sorting and aggregation; S43, the processed Python object is converted into JSON data, and the converted JSON data is stored in the database table; S44, the business processing tool and the request processor are bound, and the mapping relationship of the original JSON data, the business verification model, the Python object, the business processing tool, the converted JSON data and the database table is established.

6. The FastAPI based JSON data processing system implementation method of claim 2, wherein, Step S12 is specifically: S12a, a request processor is created in the routing package, a cryptography library is imported in the request processor, a symmetric encryption key is generated using the Fernet class of the cryptography library, and a key object is obtained; S12b, the original JSON data is received through the request processor, and the original JSON data is encrypted based on the key object to obtain encrypted binary data; S12c, the encrypted binary data is encoded to obtain encoded string data; S12d, the encoded string data is converted into JSON format data to obtain encrypted JSON data.

7. The FastAPI based JSON data processing system implementation method of claim 2, wherein, Step S12 can also be: S121, a request processor is created in the routing package, and a zlib library is imported in the request processor; S122, the original JSON data is received through the request processor, and the original JSON data is compressed using the compress function of the zlib library to obtain compressed binary data; S123, the compressed binary data is encoded to obtain encoded string data; S124, the encoded string data is converted into JSON format data to obtain compressed JSON data.

8. The FastAPI based JSON data processing system implementation method of claim 5, wherein, Step S42 is specifically: S42a, a scikit-learn library is imported in the business processing tool, and a data mining and analysis function is created based on the scikit-learn library; S42b, the data mining function is called to perform data mining on the Python object, and the data analysis function is called to perform data analysis on the Python object after data mining; S42c, based on the business processing tool, the business logic of the Python object after data analysis is processed, including calculation, analysis, filtering, sorting and aggregation.

9. The FastAPI based JSON data processing system implementation method of claim 1, wherein, Also includes S6, a visualization tool is created in the working directory, and the converted JSON data is visualized through the visualization tool to generate charts and reports.

Images