Data processing method and device based on two-stage architecture, electronic equipment and medium
By circulating algorithm data in the first architecture and distributing it to the second architecture, combined with the update of preset files, the unified management and operation and maintenance of algorithm data is realized, and the increase in management and operation and maintenance workload is solved and The problem of reduced correlation improves the correlation and management efficiency of algorithm data.
Patent Information
- Application Number
- CN202411124201.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, as the brand and version of network model algorithms increase, the workload of management and operation and maintenance has increased greatly, and the correlation between data of different algorithms has decreased, resulting in the problem of increasing management and operation and maintenance workload and reducing correlation. .
Using a two-level architecture data processing method, by flowing the algorithm data in the first architecture, obtaining the second algorithm data, it is sent to a lower-level second architecture to deploy and receive the second architecture. The uploaded preset file updates the second algorithm data, and finally sends the updated data to the second architecture to realize the unified management and operation and maintenance of the algorithm data.
It effectively reduces the management and operation and maintenance workload of algorithm data, improves the correlation between different algorithm data in the application process, and realizes the unified management and update process of algorithm data of different brands and versions.
Smart Images

Figure CN118964372A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data management technology, and in particular to a data processing method and device, electronic equipment and medium based on a two-level architecture. Background Art
[0002] In the related technologies, with the development of network model algorithms, the brands, versions and quantities of network model algorithms are increasing. At present, different management methods are used for network model algorithm data of different brands or versions, which greatly increases the management and operation workload of network model algorithm data, and also reduces the degree of correlation between different network model algorithm data in the application process.
[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a data processing method and device, electronic device and medium based on a two-level architecture, which can effectively reduce the management and operation and maintenance workload of algorithm data and improve the correlation between different algorithm data.
[0005] To achieve the above objective, an embodiment of the present application provides a data processing method based on a two-level architecture, the method is applied to a first architecture, and the method includes the following steps:
[0006] Acquire several pieces of first algorithm data;
[0007] Flow processing is performed on all the first algorithm data to obtain second algorithm data;
[0008] The second algorithm data is sent to a corresponding second architecture; the second architecture is used to deploy the second algorithm data to a preset platform; the level of the first architecture is higher than the level of the second architecture;
[0009] receiving a preset file uploaded by the second architecture, wherein the preset file is used to record SQL change statement information;
[0010] Update the second algorithm data according to the preset file to obtain third algorithm data;
[0011] The third algorithm data is sent to the second architecture, and the second architecture updates the algorithm data in the preset platform according to the third algorithm data.
[0012] In some embodiments, the processing of all the first algorithm data to obtain the second algorithm data includes:
[0013] The first algorithm data is mirrored in sequence to obtain a plurality of algorithm mirror data;
[0014] Encapsulating the algorithm image data in sequence to obtain a plurality of algorithm packages;
[0015] The algorithm packages are evaluated in sequence to obtain the second algorithm data.
[0016] In some embodiments, when the first algorithm data is mirrored in sequence, the method further includes the following steps:
[0017] Get the underlying logic information;
[0018] The first algorithm data that fails in mirroring is processed according to the fallback logic information.
[0019] In some embodiments, the algorithm image data is encapsulated in sequence to obtain a plurality of algorithm packages, including:
[0020] Write deployment scripts according to the algorithm code and algorithm name of the algorithm image data in turn;
[0021] According to the deployment script, the algorithm image data and the algorithm information in the first algorithm data are encapsulated to obtain a plurality of algorithm packages.
[0022] In some embodiments, updating the second algorithm data according to the preset file to obtain the third algorithm data includes:
[0023] Parsing the change event from the preset file to obtain the change event, wherein the change event includes an event type and a change content;
[0024] The second algorithm data is updated according to the change event to obtain the third algorithm data.
[0025] In some embodiments, the parsing the change event from the preset file includes:
[0026] Obtaining binary data in the preset file;
[0027] The change event is generated according to the binary data conversion, and the data of the change event includes structured data.
[0028] In some embodiments, updating the second algorithm data according to the change event includes:
[0029] Triggering an algorithm change task according to the change event;
[0030] Obtaining dependencies and priorities of the algorithm change tasks;
[0031] The second algorithm data is updated according to the dependency relationship and priority.
[0032] To achieve the above object, another aspect of an embodiment of the present application provides a data processing device based on a two-level architecture, the device is applied to a first architecture, and the device includes:
[0033] The first module is used to obtain a plurality of first algorithm data;
[0034] A second module is used to perform flow processing on all the first algorithm data to obtain second algorithm data;
[0035] The third module is used to send the second algorithm data to the corresponding second architecture; the second architecture is used to deploy the second algorithm data to a preset platform; the level of the first architecture is higher than the level of the second architecture;
[0036] A fourth module is used to receive a preset file uploaded by the second architecture, where the preset file is used to record SQL change statement information;
[0037] A fifth module is used to update the second algorithm data according to the preset file to obtain third algorithm data;
[0038] The sixth module is used to send the third algorithm data to the second architecture, and the second architecture updates the algorithm data in the preset platform according to the third algorithm data.
[0039] To achieve the above object, another aspect of an embodiment of the present application provides an electronic device, including:
[0040] at least one processor;
[0041] at least one memory for storing at least one program;
[0042] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0043] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0044] The embodiments of the present application include at least the following beneficial effects: The present application provides a data processing method and device, electronic device and medium based on a two-level architecture. The scheme obtains second algorithm data by circulating and processing a plurality of algorithm data in a first architecture, and then sends the second algorithm data to a second architecture of a lower level, so as to deploy the second algorithm data to a preset platform through the second architecture, and after receiving a preset file uploaded by the second architecture for recording SQL change statement information, the second algorithm data is updated based on the preset file to obtain third algorithm data, and then the third algorithm data is sent to the second architecture, so as to update the algorithm data in the preset platform according to the third algorithm data through the second architecture, thereby realizing the unified management and update process of algorithm data of different brands and versions, greatly reducing the management and operation and maintenance workload of algorithm data, and improving the correlation between different algorithm data in the application process. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 is a flow chart of a data processing method based on a two-level architecture provided in an embodiment of the present application;
[0046] Figure 2 It is a schematic diagram of a two-level architecture scenario provided in an embodiment of the present application;
[0047] Figure 3 yes Figure 1 Flow chart of step S120 in FIG.
[0048] Figure 4 yes Figure 3 Flow chart of step S310 in FIG.
[0049] Figure 5 yes Figure 3 Flow chart of step S320 in FIG.
[0050] Figure 6 yes Figure 1 Flow chart of step S150 in FIG.
[0051] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.
[0052] Figure 8 It is an application schematic diagram of the data processing method based on the two-level architecture provided in an embodiment of the present application;
[0053] Fig. 9 is a structural diagram of a data processing device based on a two-level architecture provided in an embodiment of the present application;
[0054] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.
[0056] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".
[0057] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.
[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0059] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:
[0060] Architecture, also known as software architecture, is an abstract description of the overall structure and components of software, used to guide the design of various aspects of large software systems. Architecture Description Language (ADL) is used to describe the architecture of software.
[0061] An algorithm is an accurate and complete description of a solution to a problem, and a series of clear instructions for solving a problem. An algorithm represents a strategy mechanism for describing a problem in a systematic way. In other words, an algorithm can obtain the required output within a limited time for a certain standard input. If an algorithm is flawed or unsuitable for a problem, executing the algorithm will not solve the problem. Different algorithms may use different amounts of time, space, or efficiency to complete the same task. The quality of an algorithm can be measured by space complexity and time complexity.
[0062] Mirroring is a type of redundancy. A mirror is a file storage format that can make many files into a mirror file. When it is placed on a disk with programs such as GHOST and opened with software such as GHOST, it can be restored to many files. RAID1 and RAID10 use mirroring. Common image file formats include ISO, BIN, IMG, TAO, DAO, CIF, and FCD. Mirror files not only have the "synthesis" function of ZIP compression packages, but their most important feature is that they can be recognized by specific software and can be directly burned to a disc. In fact, the image file in the usual sense can be expanded, and more information can be included in the image file. For example, system files, boot files, partition table information, etc., so that the image file can contain all the information of a partition or even a hard disk.
[0063] Binlog is a binary log file that records all DML operations of MySQL. For example, SQL statements that change database tables and content will be recorded in binlog, but queries on database tables and other content will not be stimulated. The content in binlog cannot be viewed using commands of text viewing tools (such as cat, vi), and needs to be parsed and viewed using mysqlbinlog. Data recovery, incremental backup, master-master replication, and master-slave replication can be performed through binlog log files.
[0064] ROW represents a row of records in a data table in the database. It is the basic unit of data and stores a group of related data. Just like in a table of student information, each row represents all the information of a student, including name, student number, age, etc. Such a row of data is a row in the database. Each row is composed of multiple columns, and each column represents a type of data. For example, in a student information table, there may be various columns such as name, student number, age, etc. The data in each column of each row is interrelated and together describe the attributes of a specific entity.
[0065] MySQL is a relational database management system. The SQL language it uses is the most commonly used standard language for accessing databases. Due to its small size, fast speed, and open source, MySQL is generally chosen as the website database for the development of small and medium-sized websites.
[0066] Hyper Text Transfer Protocol over Secure Socket Layer (HTTPS) is a transmission protocol for secure communication over computer networks. The main functions of HTTPS can be divided into two categories: one is to establish an information security channel to ensure the security of data transmission; the other is to confirm the authenticity of the website. For any website that uses https, you can click the lock icon in the browser address bar to view the real information of the website after authentication, or you can query it through the security signature issued by the CA organization.
[0067] The open source stream processing platform (Kafka) is a high-throughput distributed publish-subscribe messaging system. Kafka can handle all the action streams of consumers on the website. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern network. This data is usually solved by processing logs and log aggregation due to throughput requirements. For log data and offline analysis systems like Hadoop, but with the limitation of real-time processing, Kafka is a viable solution. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time messages through clusters.
[0068] Core is a new open source and cross-platform development platform that includes important features such as automatic memory management and modern high-level development languages, making it easy to develop statically typed, high-quality applications for multiple platforms.
[0069] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0070] In related technologies, with the increase of data in various industries such as education and security, the algorithm data for object recognition, tracking, and scene application based on these data is also increasing. The current number of algorithms is managed based on the rules of each target object, and there is no unified management and operation and maintenance standard, which makes it impossible to associate and apply the processing data between different algorithm data. As the amount and type of algorithm data continue to increase, the corresponding amount of association and operation and maintenance data also increases sharply, thereby increasing the association and operation and maintenance workload of algorithm data to a certain extent.
[0071] In view of this, an embodiment of the present application provides a data processing method and device, an electronic device and a medium based on a two-level architecture. After a plurality of first algorithm data of different types are circulated and processed to obtain second algorithm data, the second algorithm data is sent to the corresponding second architecture, thereby realizing a unified management process of different algorithm data. After the second algorithm data is deployed in the second architecture, a preset file recording SQL change statement information uploaded by the second architecture is received. After the second algorithm data is updated according to the preset file to obtain third algorithm data, the third algorithm data is sent to the second architecture, so that the second architecture updates the algorithm data in the preset platform according to the third algorithm data, thereby realizing a unified operation and maintenance process of the algorithm data, thereby effectively reducing the management and operation and maintenance workload of the algorithm data, and improving the degree of correlation between different algorithm data in the application process.
[0072] The data processing method based on the two-level architecture provided in the embodiment of the present application relates to the field of data management technology. The data processing method based on the two-level architecture provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements a data processing method based on a two-level architecture, etc., but is not limited to the above forms.
[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0074] Figure 1 is an optional flow chart of a data processing method based on a two-level architecture provided in an embodiment of the present application. In an embodiment of the present application, Figure 1 The method shown can be applied to Figure 2 In the two-level architecture scenario shown in Figure 1, it can be understood that Figure 2 The scenario shown includes a first architecture (group node) and at least one second architecture (provincial node). The first architecture is used to uniformly manage all algorithm data, and to distribute and deploy the algorithm data in the architecture to the corresponding second architecture. At the same time, the second architecture reports the data that needs to be changed to the first architecture, so as to uniformly manage and operate the algorithm data in the first architecture, thereby improving the correlation between different algorithm data and reducing the operation and maintenance workload of the algorithm data.
[0075] by Figure 1 The method shown is applied to Figure 2 Taking the first architecture as an example, Figure 1 The method may include but is not limited to steps S110 to S160:
[0076] Step S110, obtaining a plurality of first algorithm data;
[0077] Step S120, performing flow processing on all first algorithm data to obtain second algorithm data;
[0078] Step S130: Send the second algorithm data to the corresponding second architecture, wherein the level of the first architecture is higher than the level of the second architecture;
[0079] Step S140: receiving a preset file uploaded by the second architecture, wherein the preset file is used to record SQL change statement information;
[0080] Step S150, updating the second algorithm data according to the preset file to obtain the third algorithm data;
[0081] Step S160: Send the third algorithm data to the second architecture, so that the second architecture updates the algorithm data in the preset platform according to the third algorithm data.
[0082] In the embodiments of the present application, Figure 3 As shown, the process of transferring all the first algorithm data to obtain the second algorithm data includes but is not limited to steps S310 to S330:
[0083] Step S310, mirroring the first algorithm data in sequence to obtain a plurality of algorithm mirror data;
[0084] Step S320, encapsulating the algorithm image data in sequence to obtain a plurality of algorithm packages;
[0085] Step S330: Evaluate the algorithm packages in sequence to obtain second algorithm data.
[0086] Understandably, Figure 3 The processes shown are all executed in the first architecture. Among them, the first architecture is provided with a unique algorithm warehouse, which is responsible for the integration, issuance, deployment and other operations of the algorithm data, and provides the algorithm data in a mirrored manner during the call of the algorithm data. In this embodiment, when the first architecture performs flow processing on the first algorithm data, it mainly includes operations such as mirror production, mirror packaging, mirror evaluation of algorithm data, algorithm data shelving, algorithm data review, and algorithm data issuance and deployment. Among them, the first architecture has a sequential order between all sub-steps in the execution of the flow processing process, and the triggering of each sub-step requires a dependency relationship before and after.
[0087] In the embodiment of the present application, when executing step S310, if Figure 4 As shown, the method of the embodiment of the present application also includes but is not limited to steps S410 to S420:
[0088] Step S410, obtaining the backup logic information;
[0089] Step S420: Process the first algorithm data that fails in mirroring according to the backup logic information.
[0090] It is understandable that the fallback logic is used to process the specified task after a preset time. Among them, the delay duration in the fallback logic can be the duration of T+N, where T represents the execution cycle of the specified task and N represents the time constant set by the designer. For example, when the execution cycle T is 2 days and the time constant N is 3 days, the delay duration is 2+3 days. This embodiment delays the processing of the algorithm data that fails the mirror processing by setting the fallback logic information, thereby releasing the resources for processing the algorithm data in time, thereby improving the processing efficiency of the algorithm data that successfully processes the mirror processing.
[0091] At the same time, since each sub-step of the algorithm data flow processing in this embodiment is started only after being triggered by the previous step, this embodiment will not start the subsequent processing flow for the algorithm data that fails the mirror processing, thereby improving the utilization rate of the algorithm data processing resources.
[0092] In an embodiment of the present application, the process of making algorithm data can be to specify a physical framework, an adaptation platform, an algorithm code, or an algorithm mirror content for making. Among them, the process of making multiple algorithm data is performed asynchronously. For example, assuming that algorithm data 1, algorithm data 2, and algorithm data 3 need to be mirrored, and when the mirror data of algorithm data 1 is being made, the status of algorithm data 1 is being made, and algorithm data 2 and algorithm data 3 are in a queue. When algorithm data 1 is processed and the result of the completion of the production is returned, algorithm data 2 and algorithm data 3 are mirrored in turn.
[0093] In the embodiments of the present application, Figure 5 As shown, the process of encapsulating the algorithm image data in sequence to obtain a plurality of algorithm packages includes but is not limited to step S510 and step S520:
[0094] Step S510: Write a deployment script according to the algorithm code and algorithm name of the algorithm image data in sequence;
[0095] Step S520: Encapsulate the algorithm image data and the algorithm information in the first algorithm data according to the deployment script to obtain a plurality of algorithm packages.
[0096] It is understandable that, after the image is produced, the algorithm image data corresponding to the algorithm data is encapsulated in this embodiment to facilitate the call of the subsequent application process. Among them, this embodiment can write a deployment script corresponding to the algorithm image data based on the basic contents such as the algorithm code and the algorithm name, and then process the algorithm image data and the algorithm information into a corresponding algorithm package. In this embodiment, when encapsulating the algorithm image data, a fallback logic can also be set to process the algorithm image data that failed to be encapsulated, so as to delay the processing of the algorithm image data that failed to be encapsulated, thereby improving the utilization rate of the encapsulation resources. In this embodiment, if the algorithm image data is encapsulated successfully, the packaging logic of the successfully encapsulated algorithm package is immediately arranged.
[0097] In an embodiment of the present application, after the encapsulation of the algorithm image data is completed, the successfully encapsulated algorithm packages are predicted or evaluated in turn. It is understandable that the evaluation samples can be obtained by sampling multiple algorithm packages, and then the evaluation samples are put online for service, the image prediction is performed during the service process, and then the algorithm service is offline based on the prediction results. During the evaluation process, the task triggering process is changed with the help of binlog, so that it can be judged whether the corresponding algorithm package can pass the evaluation based on the information in the auxiliary binlog file. The algorithm package that passes the evaluation is used as the second algorithm data.
[0098] After the evaluation is completed, the second algorithm data can be sent to the second architecture. It can be understood that the first architecture of this embodiment uniformly manages the algorithm package and algorithm information of the second algorithm data, and sends and calls back the results through https+kafka information. After the successful sending, the second algorithm data is deployed to the corresponding preset platform by the second architecture, and the deployment status is synchronized to the first architecture, so as to ensure the consistency of the status of the algorithm data in the two-level architecture.
[0099] In the embodiments of the present application, Figure 6 As shown, the process of updating the second algorithm data according to the preset file to obtain the third algorithm data includes but is not limited to step S610 and step S620:
[0100] Step S610: parse and obtain a change event from a preset file, wherein the change event includes an event type and a change content;
[0101] Step S620: Update the second algorithm data according to the change event to obtain third algorithm data.
[0102] In an embodiment of the present application, the preset file may be a file that records the change event information, and the file format may be a format that can be called by the corresponding database, such as ROW format. It is understandable that the change event in the preset file includes the event type and the change content. For example, the SQL statement of the second algorithm data needs to be changed in the preset platform, so as to improve the accuracy of the network information call in the preset platform. The second architecture uploads the preset file containing the change event to the first architecture, and the first architecture can change the SQL statement in the second algorithm data according to the preset file.
[0103] In the embodiment of the present application, when the preset file is a binlog file, after obtaining the binary data in the preset file, a change event can be generated according to the binary data conversion, so that the first architecture can change the SQL statement of the preset file in the database according to the change event. It can be understood that the data of the change event can use structured data to improve the efficiency of data parsing and processing.
[0104] In the embodiments of the present application, Figure 7 As shown, the process of updating the second algorithm data according to the change event includes but is not limited to steps S710 to S730:
[0105] Step S710, triggering an algorithm change task according to a change event;
[0106] Step S720: Obtain the dependency and priority of the algorithm change task;
[0107] Step S730: Update the second algorithm data according to the dependency relationship and priority.
[0108] It is understandable that when a change event is determined according to the preset file, the first architecture can execute the change task through the preset change event trigger algorithm. For example, if it is determined according to the preset file that the algorithm data 4 needs to be changed, the change event trigger algorithm corresponding to the algorithm data 4 is used to execute the current change task to improve the accuracy of the event after the update. In this embodiment, during the change process, the SQL statement in the second algorithm data can also be updated according to the dependency and priority.
[0109] In the embodiment of the present application, in order to effectively and quickly execute Figure 1 The method shown in the figure can be implemented by setting the following components in the first architecture:
[0110] Data parser: used to parse MySQL Binlog logs in real time and extract data change events.
[0111] Task scheduler: triggers corresponding algorithm change tasks based on the parsed change events.
[0112] Task executor: changes the task execution logic according to the algorithm and feeds the results back to the task scheduler.
[0113] Dependency management module: manages the dependencies between multiple algorithm change tasks to ensure that multiple algorithm change tasks are executed in sequence.
[0114] The following is a detailed explanation of each part of the first architecture:
[0115] The data parser, including the Binlog parser, is responsible for real-time monitoring of MySQL Binlog and parsing database change events from the MySQL Binlog file. Specifically, the parsing process can be implemented using open source tools such as `mysql-binlog-connector-java`. The data processing process of the data parser includes but is not limited to the following steps:
[0116] Data capture: Connect to MySQL server and subscribe to Binlog events.
[0117] Event parsing: Parse the binary data in the Binlog file into structured change events, such as insert, update, and delete operations.
[0118] The task scheduler is used to receive the change events sent by the Binlog parser in real time and trigger the corresponding tasks according to the event type and content. The task scheduler needs to have the functions of task queue and task distribution.
[0119] Task queue: stores tasks to be executed.
[0120] Task distribution: Distribute tasks to task executors based on their dependencies and priorities.
[0121] The task executor is used to execute the logical operation of the task and feed back the execution result to the task scheduler. The task executor can be an independent service or run in the same process as the task scheduler.
[0122] It can be understood that the task executor includes task processing and result feedback:
[0123] Task processing: Execute different processing logic according to the type of task, such as calling external APIs, updating databases, etc.
[0124] Result feedback: Feedback the task execution result (success or failure) to the task scheduler.
[0125] The dependency management module is used to manage the dependencies between tasks and ensure that the tasks are executed in the correct order. For example, some tasks may depend on the completion of other tasks. It can be understood that the data processing of the dependency management module includes dependency definition and dependency checking:
[0126] Dependency definition: Define the dependencies between tasks.
[0127] Dependency checking: Check whether the tasks it depends on have been completed before executing the task.
[0128] In the embodiments of the present application, Figure 8 The application architecture shown includes a first architecture (core side) and three second architectures (provincial side). The first architecture includes a unified authentication module, a Galaxy module, a development platform (core) and an AI platform. Among them, the unified authentication module is used to uniformly authenticate system operation and maintenance personnel and algorithm developers. Only the accounts or terminal devices of authenticated system operation and maintenance personnel and algorithm developers can access and modify the first architecture. The Galaxy module is used to perform mirroring operations, mirror packaging, algorithm prediction, shelf review, algorithm distribution and algorithm installation on algorithm data. The AI platform is used to collect data sets of preset files, annotate data, model development, model training, model evaluation, engineering, mirror management, algorithm distribution and other operations. The development platform is used to perform operations such as algorithm coding, mirror management, access specifications, etc. After the system operation and maintenance personnel or algorithm developers log in through an account or enter the first architecture through a terminal device, they can perform mirroring, mirror packaging, mirror prediction, shelf review or distribution of algorithm data on existing algorithm data through corresponding operations, and can also perform algorithm coding to obtain new algorithm data. At the same time, the access process of the second architecture can also be standardized to improve the operating stability and security of the first architecture.
[0129] It is understandable that each second architecture includes a Galaxy Province side interface adapter module, a development platform and an AI platform. Among them, the Galaxy Province side interface adapter module is used to adapt to the Galaxy module in the first architecture, so that the data interaction process between the first architecture and the second architecture can be realized. The development platform is used to perform operations such as status reporting, algorithm installation, algorithm deactivation and algorithm uninstallation. When the state of the algorithm data in the second architecture changes, the preset file of the state change can be reported to the first architecture through the Galaxy Province side interface adapter module, so that the first architecture updates the algorithm data according to the change event of the preset file, and sends the update result to the first architecture at the same time, so as to achieve consistency in the state of the algorithm data between the first architecture and the second architecture. The AI platform is used to perform operations such as algorithm reception, algorithm installation, algorithm deactivation, algorithm uninstallation, status reporting, reasoning services, etc. For example, when the second architecture needs to deactivate algorithm data 5, the deactivation operation of algorithm data 5 can be controlled through the AI platform, and the change event of the deactivated algorithm data 5 can be reported to the first architecture in the form of a preset file through the Xinghe Province side interface adapter module. The first architecture can update the status of algorithm data 5 according to the change event in the preset file, and send the update result to the second architecture, so that the second architecture will also make the status of algorithm data 5 more consistent with the data status in the first architecture, thereby ensuring the consistency of the status of the same algorithm data in the two-level architecture. For example, if the second architecture needs to uninstall algorithm data 6, the AI platform and development platform in the second architecture will both uninstall algorithm data 6, and upload the corresponding preset file to the first architecture. The first architecture will perform corresponding algorithm data updates according to the preset file, and the second architecture will also be synchronously updated according to the status issued by the first architecture, thereby ensuring the consistency of algorithm data in the two-level architecture.
[0130] In the embodiments of the present application, the method implemented in the present application can be applied to target recognition scenarios, view data collaborative association scenarios, massive data efficient management scenarios, algorithm model dynamic matching scenarios, etc. In the application process, the two-section architecture mode can be used to uniformly manage and operate the corresponding algorithm data, thereby reducing the management and operation workload of the algorithm data, and realizing the mutual application process between different algorithm data.
[0131] Reference Fig. 9 , an embodiment of the present application provides a data processing device based on a two-level architecture, the device is applied to a first architecture, the device includes:
[0132] The first module 910 is used to obtain a plurality of first algorithm data;
[0133] The second module 920 is used to perform flow processing on all first algorithm data to obtain second algorithm data;
[0134] The third module 930 is used to send the second algorithm data to the corresponding second architecture; wherein the second architecture is used to deploy the second algorithm data to a preset platform; the level of the first architecture is higher than the level of the second architecture;
[0135] The fourth module 940 is used to receive a preset file uploaded by the second architecture, wherein the preset file is used to record SQL change statement information;
[0136] A fifth module 950 is used to update the second algorithm data according to a preset file to obtain third algorithm data;
[0137] The sixth module 960 is used to send the third algorithm data to the second architecture, wherein the second architecture updates the algorithm data in the preset platform according to the third algorithm data.
[0138] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0139] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data processing method based on the two-level architecture when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0140] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0141] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0142] The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0143] The memory 1020 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1020 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1020, and the processor 1010 calls and executes the data processing method based on the two-level architecture of the embodiment of this application;
[0144] Input / output interface 1030, used to implement information input and output;
[0145] The communication interface 1040 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0146] Bus 1050 , which transmits information between various components of the device (e.g., processor 1010 , memory 1020 , input / output interface 1030 , and communication interface 1040 );
[0147] The processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 are connected to each other in communication within the device via a bus 1050 .
[0148] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned data processing method based on the two-level architecture.
[0149] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0150] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0151] The embodiments of the present application provide a data processing method and device, electronic device and medium based on a two-level architecture. The present application adopts a two-level architecture model and sets four levels of computing power (parser, task scheduler, task executor and dependency management module) in the first architecture to manage and operate multiple algorithm data, thereby realizing integrated management and standardized operation of multiple algorithm data, effectively reducing the management and operation workload of algorithm data.
[0152] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0153] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0154] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0155] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0156] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0157] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0158] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0159] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0162] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A data processing method based on a two-level architecture, characterized in that: The method is applied to a first architecture and comprises the following steps: Acquire several pieces of first algorithm data; Flow processing is performed on all the first algorithm data to obtain second algorithm data; The second algorithm data is sent to a corresponding second architecture; the second architecture is used to deploy the second algorithm data to a preset platform; the level of the first architecture is higher than the level of the second architecture; receiving a preset file uploaded by the second architecture, wherein the preset file is used to record SQL change statement information; Update the second algorithm data according to the preset file to obtain third algorithm data; The third algorithm data is sent to the second architecture, and the second architecture updates the algorithm data in the preset platform according to the third algorithm data.
2. The method according to claim 1, characterized in that: The step of performing flow processing on all the first algorithm data to obtain second algorithm data includes: The first algorithm data is mirrored in sequence to obtain a plurality of algorithm mirror data; Encapsulating the algorithm image data in sequence to obtain a plurality of algorithm packages; The algorithm packages are evaluated in sequence to obtain the second algorithm data.
3. The method according to claim 2, characterized in that When the first algorithm data are mirrored in sequence, the method further comprises the following steps: Get the underlying logic information; The first algorithm data that fails in mirroring is processed according to the fallback logic information.
4. The method according to claim 2, characterized in that: The algorithm image data is encapsulated in sequence to obtain a plurality of algorithm packages, including: Write deployment scripts according to the algorithm code and algorithm name of the algorithm image data in turn; According to the deployment script, the algorithm image data and the algorithm information in the first algorithm data are encapsulated to obtain a plurality of algorithm packages.
5. The method according to claim 1, characterized in that The updating of the second algorithm data according to the preset file to obtain the third algorithm data includes: Parse the change event from the preset file to obtain the change event, wherein the change event includes an event type and a change content; The second algorithm data is updated according to the change event to obtain the third algorithm data.
6. The method according to claim 5, characterized in that The step of parsing the change event from the preset file includes: Obtaining binary data in the preset file; The change event is generated according to the binary data conversion, and the data of the change event includes structured data.
7. The method according to claim 5, characterized in that The updating of the second algorithm data according to the change event includes: Triggering an algorithm change task according to the change event; Obtaining dependencies and priorities of the algorithm change tasks; The second algorithm data is updated according to the dependency relationship and priority.
8. A data processing device based on a two-level architecture, characterized in that: The device is applied to a first architecture, and the device includes: The first module is used to obtain a plurality of first algorithm data; A second module is used to perform flow processing on all the first algorithm data to obtain second algorithm data; The third module is used to send the second algorithm data to the corresponding second architecture; the second architecture is used to deploy the second algorithm data to a preset platform; the level of the first architecture is higher than the level of the second architecture; A fourth module is used to receive a preset file uploaded by the second architecture, where the preset file is used to record SQL change statement information; A fifth module is used to update the second algorithm data according to the preset file to obtain third algorithm data; The sixth module is used to send the third algorithm data to the second architecture, and the second architecture updates the algorithm data in the preset platform according to the third algorithm data.
9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
