An embedded software research and development whole life cycle data acquisition and management system
The system for automated acquisition and full-link correlation analysis of multi-source heterogeneous data has solved the problems of data dispersion and heterogeneous toolchains in embedded software development, realizing data transparency and rapid traceability throughout the entire lifecycle, and improving R&D efficiency and quality control.
Patent Information
- Application Number
- CN202610509274.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-17
- Publication Date
- 2026-07-17
AI Technical Summary
In the process of embedded software development, data dispersion and complex heterogeneous toolchains lead to difficulties in data integration, insufficient real-time performance, and low analysis efficiency, making it difficult to achieve full-chain automated traceability and efficiency quantification, thus affecting development efficiency and quality assurance.
Design a system for automated acquisition and end-to-end correlation analysis of multi-source heterogeneous data, including data acquisition, storage, computing and application layers, supporting real-time data processing and offline analysis, providing multi-role collaborative visualization and risk warning, and having the ability to expand data source types and configure analysis models.
It enables transparency in the embedded software development process and rapid problem tracing, solves the data silo problem, improves development efficiency and quality control, and supports data-driven decision-making and process optimization throughout the entire lifecycle.
Smart Images

Figure CN122411897A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of embedded software development technology, and more specifically, to a data acquisition and management system for the entire lifecycle of embedded software development. Background Technology
[0002] Current embedded software development faces challenges such as expanding development scale, fragmented data throughout the development process, difficulty in intuitively reflecting development progress, and the complexity of heterogeneous toolchains in embedded software development. The entire lifecycle of embedded software development involves multiple stages, including requirements gathering, design, development, testing, and deployment. Data sources during the development process include requirements gathering, code repositories, continuous integration and continuous delivery pipelines, and defect tracking systems. The diverse data formats (such as JSON, log text, and database tables) lead to difficulties in data integration, insufficient real-time performance, and low analysis efficiency, resulting in a black-box approach to the process, delayed quality risks, and high troubleshooting costs. Currently, data collection and management throughout the software development lifecycle mainly rely on developers manually designing, monitoring, and using log collection tools. In the embedded software development lifecycle, the heterogeneous and complex toolchains make it difficult for traditional data collection methods to achieve end-to-end automated traceability and efficiency quantification, thus hindering efficient iteration and quality assurance of embedded software. Summary of the Invention
[0003] To address the aforementioned technical challenges, this invention aims to design a full lifecycle data acquisition and management method for embedded software development. It employs a combination of automated multi-source heterogeneous data acquisition and end-to-end correlation analysis, supporting real-time acquisition and unified management of data across all stages, including requirements, design, development, testing, and deployment. It also supports process visualization and risk warning based on process node monitoring and performance indicator quantification, and features scalable data source types, configurable analysis models, and customizable multi-role collaboration capabilities. This method transforms the development process from a "black box" to a "fully traceable" one, supports cross-toolchain collaboration of embedded software development tools, and improves the efficiency and quality control of embedded software development.
[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0005] An embedded software development lifecycle data acquisition and management system includes a data acquisition layer module, a data storage layer module, a data computing layer module, and a data application layer module;
[0006] The data acquisition layer module is responsible for acquiring data from multiple heterogeneous data sources and performing preliminary format standardization on the acquired data;
[0007] The data storage layer module is responsible for receiving and storing standardized data from the data acquisition layer module, and adopts a hybrid storage strategy for different types of data and access requirements;
[0008] The data computing layer module is responsible for processing and analyzing the data stored in the data storage layer, and uses distributed computing technology to achieve collaborative computing between real-time stream processing and offline batch processing;
[0009] The data application layer module applies the processed data to actual business operations, providing data-driven application services for different roles in the R&D process.
[0010] As a preferred embodiment of the present invention, the multi-source heterogeneous data source is a multi-source heterogeneous data source from the requirements, design, development, testing and deployment stages of the entire software development lifecycle; the multi-source heterogeneous data includes structured data, semi-structured data and unstructured data.
[0011] In a preferred embodiment of the present invention, the data acquisition layer module supports multiple acquisition methods, including front-end point data acquisition, database data acquisition, and log acquisition, and can adopt appropriate acquisition methods and technologies according to the data source type, specifically including:
[0012] For code commit logs from version control systems, a customized agent is used for collection; for build and deployment events generated by continuous integration (CI) / continuous deployment (CD) pipelines, they are captured in real time through the API of the integrated Jenkins tool; and for front-end user actions, a visual tracking technology is used for configurable collection.
[0013] In a preferred embodiment of the present invention, the data storage layer module contains different types of data, including structured, semi-structured, and unstructured data; access requirements include cold data and hot data; the hybrid storage strategy includes multiple storage solutions such as distributed file systems, relational databases, or object storage, and supports unified storage of structured, semi-structured, and unstructured data, with data backup and recovery capabilities.
[0014] As a preferred embodiment of the present invention, OBS is used for low-cost, high-capacity storage, Kafka / DMQ is used as a real-time data buffer, and HDFS is used to support high-performance batch processing computation.
[0015] As a preferred embodiment of the present invention, the data storage layer module adopts a hierarchical storage architecture. The real-time collected data is first buffered in a message queue to decouple the collection and calculation process. Hot data that needs to be calculated in real time is stored in an in-memory database. The raw data required for batch processing analysis is persistently stored in HDFS or OBS object storage and managed in a hot and cold tier according to the data lifecycle strategy.
[0016] As a preferred embodiment of the present invention, the data computing layer module includes multiple stages such as data preprocessing, feature extraction, model calculation, and result output; the platform implements integrated stream and batch computing based on Flink, which can process real-time data streams at the millisecond level and perform offline analysis of historical batch data; the intelligent analysis function, which uses pre-trained language models to analyze the consistency of requirement documents or uses graph algorithms to analyze code dependencies, is implemented in this layer.
[0017] In a preferred embodiment of the present invention, the data application layer module is responsible for transforming data from embedded software development into services and visual interfaces that users can directly use, providing customized workbenches for different roles, specifically:
[0018] Project leaders can view project progress and risk warnings through a global dashboard; software development engineers receive code quality feedback and collaboration conflict alerts; test engineers use test case recommendation and defect correlation analysis tools; all applications provide services through interfaces or APIs, supporting deep integration with existing R&D toolchains.
[0019] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:
[0020] 1. Full lifecycle data integration and traceability: By automatically collecting data from requirements, code, testing, deployment and other stages, a complete data chain of "requirements-code-testing-deployment" is built to achieve transparency in the R&D process and rapid traceability of problems. For example, online failures can be accurately located to specific code submissions and test omissions.
[0021] 2. Integrated Acquisition and Governance of Multi-Source Heterogeneous Data: Supports unified acquisition of structured, semi-structured and unstructured data. Through adapters, it integrates data from multiple toolchains such as code repositories, CI / CD pipelines, and defect tracking, solving the problems of data silos and protocol fragmentation, and ensuring data integrity and consistency.
[0022] 3. Data processing and intelligent analysis of batch processing: Real-time monitoring and offline analysis are achieved through a unified batch processing architecture. Multiple analysis methods are used to analyze and visualize requirements documents, code quality, defect distribution, etc., supporting risk prediction and performance optimization.
[0023] 4. Scenario-based data application and process embedding: Through tools such as visual dashboards, intelligent early warning, and performance attribution analysis, data capabilities are embedded into all aspects of requirements assessment, code development, test execution, deployment and maintenance, to achieve data-driven R&D decision-making and automated process optimization. Attached Figure Description
[0024] Figure 1This is a schematic diagram of the architecture of an embedded software development full lifecycle data acquisition and management system in this embodiment;
[0025] Figure 2 This is a flowchart of a data acquisition and management system for the entire lifecycle of embedded software development in this embodiment. Detailed Implementation
[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0027] The embodiments provided by the present invention will be described in detail below:
[0028] like Figure 1 The diagram shows an architecture of an embedded software development lifecycle data acquisition and management system, comprising a data acquisition layer, a data storage layer, a data computation layer, and a data application layer. Figure 2 As shown, the data acquisition layer is responsible for automatically collecting data from various data sources throughout the entire R&D process; the data storage layer is responsible for the unified storage and management of the massive amounts of multi-source heterogeneous data collected; the data computing layer is responsible for processing, analyzing, and intelligently computing the data in the storage layer; and the data application layer transforms the computing results into specific services and tools that support R&D decision-making, process optimization, quality control, and compliance auditing. Through the organic collaboration of these layers, data-driven, refined management of the entire lifecycle of embedded software R&D is achieved.
[0029] As a preferred embodiment, the data acquisition layer automatically collects multi-source heterogeneous data (such as requirement documents, code commit logs, test case execution results, deployment pipeline information, runtime monitoring data, etc.) generated during the embedded software development process. This layer employs multiple methods, including front-end tracking, log collection tools, and database synchronization components, to ensure data integrity and real-time performance. The collected results are then sent to the data storage layer.
[0030] In a preferred embodiment, the data storage layer is responsible for receiving and storing data from the acquisition layer. A hybrid storage strategy is adopted to address different types of data (structured, semi-structured, unstructured) and access requirements (cold data, hot data). For example, OBS is used for low-cost, high-capacity storage, Kafka / DMQ is used as a real-time data buffer, and HDFS is used to support high-performance batch processing computation, thus achieving a balance between cost, performance, and reliability.
[0031] In a preferred embodiment, the data computation layer is the core processing engine of the platform, responsible for performing calculations and analyses on the data in the storage layer. This layer possesses basic computing capabilities (such as multi-source data fusion and integrated stream and batch processing), intelligent analysis capabilities (such as NLP-based requirement document analysis and machine learning-based defect prediction), and scenario-based computing capabilities (such as calculating core indicators for each stage of requirements, development, testing, deployment, operation and maintenance, and decommissioning). The calculation results support upper-layer applications.
[0032] As a preferred embodiment, the data application layer provides data-driven application services to different roles in the R&D process (managers, developers, testers, operations personnel, etc.). This mainly includes a full-chain visualization dashboard, R&D process optimization tools (such as requirement change impact analysis and intelligent test case recommendation), a quality and risk management system (such as code quality closed-loop management and defect lifecycle tracking), and compliance and audit support functions, ultimately realizing the transformation of data value.
[0033] To further illustrate the specific embodiments of the present invention, this embodiment further discloses:
[0034] like Figure 1 As shown, the design and construction of the data acquisition layer, data storage layer, data computation layer, and data application layer were completed first, and then deployed on a cloud-native architecture-based platform. A unified scheduling and management framework connects the entire data process from acquisition, storage, computation to application, and supports administrators in configuring and optimizing each layer's components according to actual needs. The workflow is as follows: Figure 2 As shown.
[0035] After connecting to various data sources from the embedded software development platform, the data acquisition layer calls the corresponding collector based on the data source type (such as structured databases, log files, and front-end behavioral data). For example, for code commit logs from version control systems (such as Git), a customized agent is used for collection; for build and deployment events generated by continuous integration / continuous deployment (CI / CD) pipelines, real-time capture is performed through APIs integrated with tools such as Jenkins; and for front-end user actions, configurable collection is achieved using visual tracking technology. The collected data, after initial format standardization, is sent to the data storage layer.
[0036] In this embodiment, the data storage layer adopts a hierarchical storage architecture. Real-time acquired data is first buffered in a message queue to decouple the acquisition and computation processes. Hot data that requires real-time computation can be stored in an in-memory database; raw data required for batch processing analysis is persistently stored in HDFS or OBS object storage, and cold / hot tier management is implemented according to data lifecycle strategies to balance storage costs and query efficiency.
[0037] In this embodiment, the data computation layer includes data preprocessing, feature extraction, model computation, and result output. The platform uses Flink to achieve integrated stream and batch computation, enabling millisecond-level processing of real-time data streams (such as test execution status monitoring) and offline analysis of historical batch data. Intelligent analysis functions, such as using pre-trained language models to analyze the consistency of requirement documents or using graph algorithms to analyze code dependencies, are all implemented in this layer. The computation results are pushed to the data application layer or written back to the storage layer for later use.
[0038] In this embodiment, the data application layer is responsible for transforming data from embedded software development into services and visual interfaces that users can directly use. In this example, the application layer provides customized workbenches for different roles. Project leaders can view project progress and risk warnings through a global dashboard; software development engineers can receive code quality feedback and collaboration conflict alerts; and test engineers can use test case recommendations and defect correlation analysis tools. All applications provide services through interfaces or APIs, supporting deep integration with existing development toolchains.
[0039] At this point, users can gain a comprehensive understanding of the entire lifecycle of embedded software development through the data application layer, and drive process optimization, quality improvement, and risk control based on data insights. If anomalies or areas for improvement are discovered, development practices can be adjusted, and data can be collected and analyzed again through this platform, forming a closed loop of continuous improvement.
[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A data acquisition and management system for the entire lifecycle of embedded software development, characterized in that: It includes a data acquisition layer module, a data storage layer module, a data computing layer module, and a data application layer module; The data acquisition layer module is responsible for acquiring data from multiple heterogeneous data sources and performing preliminary format standardization on the acquired data; The data storage layer module is responsible for receiving and storing standardized data from the data acquisition layer module, and adopts a hybrid storage strategy for different types of data and access requirements; The data computing layer module is responsible for processing and analyzing the data stored in the data storage layer, and uses distributed computing technology to achieve collaborative computing between real-time stream processing and offline batch processing; The data application layer module applies the processed data to actual business operations, providing data-driven application services for different roles in the R&D process.
2. The embedded software development full lifecycle data acquisition and management system according to claim 1, characterized in that: The multi-source heterogeneous data source refers to the multi-source heterogeneous data source from the requirements, design, development, testing and deployment stages of the entire software development life cycle; the multi-source heterogeneous data includes structured data, semi-structured data and unstructured data.
3. The embedded software development full lifecycle data acquisition and management system according to claim 1, characterized in that: The data acquisition layer module supports multiple acquisition methods, including front-end point data acquisition, database data acquisition, and log acquisition. It can also employ appropriate acquisition methods and technologies based on the data source type, specifically including: For code commit logs from version control systems, a customized agent is used for collection; for build and deployment events generated by continuous integration (CI) / continuous deployment (CD) pipelines, they are captured in real time through the API of the integrated Jenkins tool; and for front-end user actions, a visual tracking technology is used for configurable collection.
4. The embedded software development full lifecycle data acquisition and management system according to claim 1, characterized in that: The data storage layer module contains different types of data, including structured, semi-structured, and unstructured data; access requirements include cold data and hot data; the hybrid storage strategy includes using multiple storage solutions such as distributed file systems, relational databases, or object storage, and supports unified storage of structured, semi-structured, and unstructured data, with data backup and recovery capabilities.
5. The embedded software development full lifecycle data acquisition and management system according to claim 4, characterized in that: Use OBS for low-cost, high-capacity storage, Kafka / DMQ as a real-time data buffer, and HDFS to support high-performance batch processing computation.
6. The embedded software development full lifecycle data acquisition and management system according to claim 4, characterized in that: The data storage layer module adopts a hierarchical storage architecture. Real-time collected data is first buffered in a message queue to decouple the collection and calculation process. Hot data that needs to be calculated in real time is stored in an in-memory database. Raw data required for batch processing analysis is persistently stored in HDFS or OBS object storage and managed in hot and cold tiers according to the data lifecycle strategy.
7. The embedded software development full lifecycle data acquisition and management system according to claim 1, characterized in that: The data computation layer module includes multiple stages such as data preprocessing, feature extraction, model computation, and result output. The platform is based on Flink to realize integrated stream and batch computing, which can process real-time data streams at the millisecond level and perform offline analysis of historical batch data. The intelligent analysis function, which uses pre-trained language models to analyze the consistency of requirement documents or uses graph algorithms to analyze code dependencies, is implemented in this layer.
8. The embedded software development full lifecycle data acquisition and management system according to claim 1, characterized in that: The data application layer module is responsible for transforming data from embedded software development into services and visual interfaces that users can directly use, providing customized workbenches for different roles, specifically: Project leaders can view project progress and risk warnings through a global dashboard; software development engineers receive code quality feedback and collaboration conflict alerts; test engineers use test case recommendation and defect correlation analysis tools; all applications provide services through interfaces or APIs, supporting deep integration with existing R&D toolchains.
