Quality standard updating system based on web crawler and automatic notification

Abstract

The invention discloses a quality standard updating system based on web crawlers and automatic notification. The system comprises the following modules: a timed task module, a manual trigger module, a QMS system interface module, a web crawler module, a standard state updating module, a database module, a subscription notification module, a nailing notification module and a principal management module. According to the technical scheme, the updating efficiency of the standard is improved, the standard can be updated regularly and immediately through the timed task and manual triggering, and delay caused by manual operation is avoided. Errors caused by manual work are reduced, the error probability caused by manual input and operation is reduced through automatic web crawler and data processing, and the accuracy and integrity of standard information are ensured. In conclusion, according to the technical scheme, by means of automation, intelligence and informatization, the efficiency and accuracy of quality standard updating management are remarkably improved.

Description

[Technical field]

[0001] The present invention relates to a quality standard updating system based on a web crawler and automatic notification, and belongs to the field of computer technology. The system realizes dynamic updating and notification of quality standards by automated means. The system covers multiple modules, including a timed task module, a manual trigger module, a QMS system interface module, a web crawler module, a standard status updating module, a database module, a subscription notification module, a DingTalk notification module, and a responsible person management module. [Background technology]

[0002] In traditional quality management systems (QMS), standard updates are a crucial link. Quality standards are usually issued by countries, industries or companies, and are constantly updated with technological advances and changes in market demand. Timely acquisition and application of the latest quality standards are essential for companies to maintain competitiveness and compliance. However, most companies currently still rely on manual methods to update standards, which has the following problems:

[0003] Inefficiency of manual updating: Manually searching, downloading, and updating quality standards requires a lot of manpower and time, and is prone to omissions or delays;

[0004] Inaccurate updates: During manual operations, inaccurate updates may occur due to negligence or misunderstanding of information.

[0005] Untimely information: Due to the periodicity of manual updates, enterprises may not be able to obtain the latest quality standards in a timely manner, thus affecting production and quality control;

[0006] Insufficient notification mechanism: Traditional methods lack an effective notification mechanism, and relevant responsible personnel cannot be informed of standard updates in a timely manner, affecting work arrangements.

[0007] In order to solve the many problems existing in the traditional quality standard update process, some automated systems based on computer technology and Internet technology have emerged. These systems improve the efficiency and accuracy of quality standard updates through digitalization and automation. However, most of these systems lack flexibility and customization and cannot adapt to the needs of different enterprises and standard updates. [Summary of the invention]

[0008] The present invention provides a quality standard update system based on web crawlers and automatic notifications, aiming to solve many problems such as low efficiency and inaccurate updates of manual updates, so as to achieve efficient and accurate automatic standard updates. The system includes the following main functional modules:

[0009] Scheduled task module: The scheduled task module is used to trigger standard update tasks on a scheduled basis, allowing users to set different update frequencies and time points to achieve automated scheduled updates.

[0010] Manual trigger module: The manual trigger module allows users to manually start standard update tasks at any time to meet update needs in emergency situations.

[0011] QMS system interface module: The QMS system interface module retrieves the standard number from the company's internal QMS system and provides a data source to support the web crawler module in capturing standard information.

[0012] Web crawler module: The web crawler module automatically crawls standard update information from the designated official website or database and parses out useful data content.

[0013] Standard status update module: The standard status update module automatically updates the standard status in the internal database according to the latest captured information and records the update log.

[0014] Database module: The database module is used to store and manage all standard information and update records, and provide efficient data query and maintenance functions.

[0015] Subscription Notification Module: The subscription notification module allows users to subscribe to update information of specific standards and configure the method and content of receiving notifications.

[0016] DingTalk Notification Module: The DingTalk Notification Module sends standard update notifications to relevant persons in charge through the DingTalk platform to ensure timely communication of information.

[0017] Responsible person management module: The relevant responsible person interface module manages the information and permissions of the responsible person, ensures that notifications are sent to the correct person, and provides a query interface.

[0018] In summary, the system can realize real-time monitoring and dynamic updating of quality standards, improve the efficiency and accuracy of quality standards management, and provide users with convenient and timely standard update solutions.

[0019] The system has the following main advantages:

[0020] Automated update: The system can automatically trigger standard update tasks at a fixed time, reducing the need for manual operations and greatly improving work efficiency.

[0021] Efficient and accurate: The latest quality standard information is automatically captured through the web crawler module to ensure the accuracy of updates and avoid errors and omissions that may occur in manual operations.

[0022] Real-time: The system can trigger standard updates regularly or manually to ensure that companies can obtain the latest quality standard information in a timely manner and respond quickly to market and technological changes.

[0023] Comprehensiveness: The QMS system interface module, web crawler module, standard status update module, database module and other functional modules are organically combined to achieve comprehensive management and updating of standard information.

[0024] Subscription and notification functions: The subscription notification module and the DingTalk notification module ensure that relevant persons in charge can receive standard update information in a timely manner, avoiding the lag in information transmission and improving work efficiency and response speed.

[0025] Data security and management: The system's built-in database module is used to store and manage all standard information and update records, providing efficient data query and maintenance functions while ensuring data security and integrity.

[0026] Flexibility and customization: The system allows users to set the frequency and time of scheduled tasks according to specific needs, and can also manually trigger update tasks, with high flexibility and customization capabilities.

[0027] Comprehensive management: The system provides a series of functions such as standard status update, update record storage, message subscription, automatic notification, etc., realizing the full life cycle management of quality standards.

[0028] Ease of use: Through user-friendly modular design, users can easily operate and manage, reducing the complexity of system use.

[0029] Efficient information transmission: Through instant messaging tools such as DingTalk, information can be quickly transmitted to relevant personnel, improving the timeliness and effectiveness of information transmission.

[0030] Improve corporate compliance: Timely acquisition and application of the latest quality standards ensures that companies can maintain competitiveness and compliance, and reduces the risks caused by untimely standard updates.

[0031] Through the above advantages, the system of the present invention can significantly improve the automation level and work efficiency of enterprises in quality standard updating and management, and provide enterprises with an efficient, accurate and intelligent solution.

Brief Description of the Drawings

[0032] Figure 1 Schematic diagram of the quality standards update system based on web crawlers and automatic notifications.

[0033] Figure 2 Schematic diagram of the standard task flow based on creating a scheduled update.

[0034] Figure 3 Diagram of the standard task flow based on creating a manual update. [Specific implementation method]

[0035] Embodiment 1: Figure 1-Figure 2As shown, the main function of the scheduled task module is to trigger the standard update task regularly to ensure the timely update of standard information. The workflow is as follows:

[0036] 1) Create a scheduled task: The administrator creates a scheduled task in the system and sets the trigger frequency and specific time of the task.

[0037] 2) Set trigger conditions: Configure the trigger conditions for scheduled tasks, such as specific time points daily, weekly, or monthly.

[0038] 3) Task triggering: When the preset time is reached, the scheduled task module automatically triggers the update task.

[0039] 4) Calling the web crawler module: The scheduled task module calls the web crawler module and starts to obtain the latest quality standard information based on the standard number in the QMS system.

[0040] 5) Update standard information: After the web crawler module obtains the latest standard information, the standard status update module is responsible for updating it to the system.

[0041] 6) Data storage: The updated standard information is stored in the database module for subsequent query and management.

[0042] 7) Generate notification information: The subscription notification module generates corresponding notification content according to the updated standard information.

[0043] 8) Send notification: The DingTalk notification module sends the generated notification information to the relevant person in charge through DingTalk to ensure that the information is conveyed in a timely manner.

[0044] Embodiment 2: Figure 1-Figure 3 As shown, in this embodiment, the quality standard update system based on web crawlers and automatic notifications implements standard updates through a manual trigger module. The main function of the manual trigger module is to allow users to manually trigger standard update tasks when needed, so as to update standard information in a timely manner under special circumstances. The specific implementation steps are as follows:

[0045] 1) Manual start by the user: The user selects to manually trigger the update task on the system interface to start the standard update process.

[0046] 2) Calling the web crawler module: The manual trigger module calls the web crawler module to start acquiring the latest quality standard information based on the standard number in the QMS system.

[0047] 3) Update standard information: After the web crawler module obtains the latest standard information, the standard status update module is responsible for updating it to the system.

[0048] 4) Data storage: The updated standard information is stored in the database module for subsequent query and management.

[0049] 5) Generate notification information: The subscription notification module generates corresponding notification content according to the updated standard information.

[0050] 6) Send notification: The DingTalk notification module sends the generated notification information to the relevant person in charge through DingTalk to ensure that the information is conveyed in a timely manner.

[0051] Through the manual update mechanism of the manual trigger module, this embodiment provides users with a flexible means to update standard information in a timely manner under special circumstances, thereby ensuring the timeliness and accuracy of the standard information.

[0052] The present invention realizes the automation and real-time updating of quality standards by designing a quality standard updating system based on web crawlers and automatic notification, overcoming the problems of inefficiency, inaccuracy and easy omission of traditional manual updating. Through the collaborative work of multiple modules, comprehensive management and efficient notification of standard information are realized, providing enterprises with an efficient, accurate and intelligent quality standard management solution.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A quality standard update system based on web crawlers and automatic notifications, the main features of which are: 1) Scheduled task module: regularly trigger standard update tasks to ensure that the system regularly checks and updates quality standards; 2) Manual trigger module: allows users to manually trigger standard update tasks to meet temporary or emergency needs; 3) QMS (Quality Management System) system interface module: retrieve the standard number from the company's QMS system as the input source for the web crawler to obtain standard information; 4) Web crawler module: According to the standard number, the crawler obtains relevant standard information to ensure the latest and accuracy of the standard information; 5) Standard status update module: Based on the acquired standard information, the standard status is updated after manual review and the update log is recorded. Based on the matching relationship between the standard data of the QMS system and the qualification data, automatic early warning of the expiration / expiration of the qualification is realized, thereby effectively assisting the change and expansion of the qualification; 6) Database module: stores updated standard status and update records to ensure data persistence and traceability; 7) Subscription notification module: Generate subscription notifications based on the updated content to ensure that relevant persons in charge can obtain standard update information in a timely manner; 8) DingTalk notification module: Send notifications through DingTalk to ensure that information can be quickly and accurately transmitted to relevant persons in charge; 9) Responsible person management module: Receive feedback and confirmation from responsible persons to ensure closed-loop management of updated information.

2. A quality standard updating system based on web crawler and automatic notification according to claim 1, characterized in that The scheduled task module is mainly used to trigger the standard update task on a scheduled basis and has the following main features: 1) Timing trigger function: can trigger standard update tasks according to preset time intervals or specific time points to achieve automatic timing updates; 2) Flexible time settings: Allow users to customize the trigger time, including daily, weekly, monthly or specific date and time, to meet the update needs of different time and cycle; 3) Multi-task management: supports simultaneous management of multiple standard update tasks, and can independently set the timing trigger rules for each task; 4) Task scheduling and priority: It has the function of task scheduling and arranges the timing trigger sequence according to the importance or priority of the task; 5) Exception handling mechanism: handles abnormal situations that may occur during the execution of scheduled tasks; 6) Execution log record: record the execution status of each scheduled task, including execution time, execution results, error information, etc.; 7) User rights management: Ensure that only authorized users can configure and manage scheduled tasks.

3. A quality standard updating system based on web crawler and automatic notification according to claim 1, characterized in that The manual trigger module has the following main features: 1) Manual start function: allows users to manually trigger the standard update task and immediately perform standard update checks. Update checks can be performed on single or multiple specified standards; 2) User interface: Provides a user-friendly interface to facilitate user operation. The interface includes functional areas such as standard list, update record query, and manual trigger button. It supports querying and filtering target standards based on standard number, name, category and other conditions. The interface displays the execution progress and results of manually triggered tasks; 3) Permission control: ensure that only authorized users can manually trigger update tasks. Role permissions control which users can manually trigger updates. Permission levels are set so that high-level users can trigger more standard updates. 4) Task management: The execution status of manually triggered tasks can be viewed on the interface. Manually triggered tasks can be aborted and re-executed. Task execution logs are recorded for auditing. 5) Notification mechanism: After the manually triggered task is completed, a notification is sent to relevant personnel. The notification method can be configured, such as DingTalk, station letter, email, message, etc. The notification content includes the standard name, update content summary, etc.; 6) User feedback: Users can provide confirmation feedback on the results of manually triggered tasks. This supports functions such as annotating result descriptions and attaching explanation files. Feedback records are stored in the database.

4. According to the quality standard updating system based on web crawler and automatic notification as described in claim 1, the QMS system interface module has the following main features: 1) Data acquisition function: connect with the company's internal QMS system to obtain the list of standard numbers that need to be checked. It supports batch acquisition and incremental acquisition to meet different data demand scenarios. It can filter and extract the required standard information according to conditions such as standard number, name, category, etc., and support pulling the latest standard data from the QMS system regularly or in real time; 2) Compatibility: Provide a unified data interface layer to shield the differences between the underlying QMS systems and support a variety of common QMS system interface protocols, such as Web Service, RESTful API, etc., to meet the company's future needs for replacing the new QMS system and ensure the scalability of the system; 3) Secure transmission: Use encrypted transmission protocols (such as HTTPS, SSH, etc.) to communicate with the QMS system to prevent data theft, support two-way identity authentication to ensure the legal identity of both systems, configurable encryption for transmitted data, and provide a variety of encryption algorithm options to protect data confidentiality; 4) Data Cache: A standard information cache mechanism is set up to reduce the query pressure on the QMS system. Automatic expiration and active update strategies of the cache are supported, and the cache expiration time is reasonably set according to the standard change frequency.

5. A quality standard updating system based on web crawler and automatic notification according to claim 1, characterized in that The web crawler module has the following main features: 1) Information crawling: Automatically crawl the target website pages according to the configured seed URL, support multiple web page traversal strategies such as breadth-first and depth-first, support two crawling modes: incremental crawling and full crawling, can crawl a large amount of web page data concurrently and efficiently, and support regular or real-time cyclic crawling; 2) Data parsing: Perform structured parsing on the captured web pages, and use XPath, regular expressions and other technologies to accurately extract the required fields. It supports parsing multiple web page formats such as HTML, XML, and JSON. The parsing fields are configurable, such as standard number, status, replacement information, etc., to eliminate invalid information and noise data in the web pages. 3) Anti-crawler mechanism: Supports user agent (UA) pool, simulates the access characteristics of different terminals, realizes dynamic switching of IP addresses, avoids banned IP addresses, sets random delays for requests, simulates the access behavior of normal users, supports verification code recognition, automatically bypasses verification code blocking, and has anti-crawling pool bypass capabilities, such as JavaScript execution, page decryption, etc. 4) Storage mechanism: The successfully captured web page data is saved in a text / NoSQL database, which supports breakpoint resuming to prevent data loss, and configures a multi-level cache strategy to improve data query efficiency; 5) Scheduling mechanism: supports multi-threaded / multi-process / distributed working modes, and automatically adjusts the crawler's working mode according to crawling tasks and website characteristics; 6) Monitoring and alarm: real-time monitoring of crawler crawling progress, speed, success rate and other indicators, configuration of warning thresholds, timely alarm notification when indicators are abnormal, support automatic or manual restart, stop and other maintenance operations; 7) Scalability: Provides a standard componentized interface to facilitate secondary development and expansion, and supports adding new parsing, anti-crawling and other functions through plug-ins or customization.

6. A quality standard updating system based on web crawler and automatic notification according to claim 1 is characterized in that The standard status update module has the following main features: 1) Data processing: Receive the original standard information data crawled by the web crawler module, perform pre-processing such as cleaning, format conversion, and duplicate removal on the original data, compare it with the records in the existing standard library, identify the changed fields, and automatically update the latest status of the standard to the database according to the changed content; 2) Rule engine: The business rule library for standard status determination is built-in. The status determination rules are pre-defined based on industry practices and expert experience. The rules are configurable and support modification and maintenance of rules through configuration files. The corresponding status determination rule sets are used for different types of standards. The combination and nesting of rules are supported to enhance the determination ability and flexibility. 3) Log Recording: A log is recorded in detail for each standard status change. The log information includes the change time, standard name, old status, new status, etc. At the same time, key evidence of the status change is recorded, such as web page URL, snapshot, etc. The log can be generated into an offline file or stored in a log database, supporting formatted output and filter query of log content; 4) Manual review: Add a manual review link for some key standard status changes that are automatically determined. The auditor will review the change evidence and decide whether to finally confirm it. It supports the addition of audit opinions to form a complete audit trail; 5) Difference notification: When a change in the standard status is found, a notification is actively generated. The notification includes standard information, summary of the changed content, etc. The notification form is configurable, such as message push, email, process, etc.

7. A quality standard updating system based on web crawler and automatic notification according to claim 1 is characterized in that The database module has the following main features: 1) Data storage: Use relational or non-relational databases to store standard information, update records, logs and other structured and unstructured data. Support the storage of standard full data and incremental data. The storage format is standardized and unified to facilitate data exchange and integration. 2) Data management: Provides functions such as adding, modifying, deleting, and querying standard information. The query supports filtering and combining based on conditions such as number, name, and category. Modification and deletion operations are subject to permission control to prevent misoperation. It manages data version information and implements data backtracking. 3) Backup and recovery: Supports full and incremental backup modes. Full backup is used for data migration and system recovery, while incremental backup saves storage space and improves backup efficiency. Backup files can be stored locally or uploaded to remote storage. It provides data recovery function to quickly rebuild the database based on the backup.

8. A quality standard updating system based on web crawler and automatic notification according to claim 1, characterized in that The subscription notification module has the following main features: 1) Subscription management: Users can subscribe to the standard updates of their interest by themselves. Subscriptions can be made by standard category, industry, etc. There is a many-to-many relationship between subscribers and standards. 2) Notification configuration: The triggering time of notifications can be configured, such as real-time, daily summary, etc. The content of notifications can be customized, such as standard overview, update details, etc., and multiple notification sending methods are supported, such as email, SMS, application push, etc.; 3) Real-time notification: When a subscribed standard is updated, the notification process is triggered immediately. Based on the subscriber configuration, the notification content is formatted and sent. The notification information must be accompanied by update details for easy review by the subscriber. 4) Notification templates: Multiple notification templates are preset, such as standard revocation and revision. The template content can be customized, including text, pictures, etc., and supports self-service creation and maintenance of customized notification templates; 5) Subscription Audit: Record information such as subscribers, subscription criteria, subscription time, etc., and save each notification record sent for auditing and data analysis.

9. A quality standard updating system based on web crawler and automatic notification according to claim 1, characterized in that The DingTalk notification module has the following main features: 1) Send notifications through the DingTalk interface. This module needs to be integrated with the DingTalk open platform to obtain access credentials (AccessToken). Using the DingTalk message push API, messages can be pushed to designated DingTalk users or groups. Push messages can be in multiple formats such as text, links, cards, etc., which can be selected according to needs. The objects that need to be notified can be automatically identified based on standard types, update impact range, etc.; 2) Support custom notification templates and provide a visual editing interface for notification templates. Users can edit the template content independently. Dynamic parameters such as standard name, update description, etc. can be embedded in the template and dynamically replaced during the actual push. It supports the configuration of different types of notification templates, such as general updates, major updates, department notifications, etc. The created templates are saved in the template library for easy reuse and management; 3) Integrate other communication tools. In addition to DingTalk, you can consider integrating other commonly used enterprise instant messaging tools, such as WeChat for Enterprise, Feishu, etc. For different communication channels, you need to adapt the message push API interface provided by them. When sending notifications, users can be supported to select a single or multiple communication channels. Notification templates can be managed in a unified manner to achieve cross-platform template reuse.

10. A quality standard updating system based on web crawler and automatic notification according to claim 1, characterized in that The responsible person management module has the following main features: 1) User management: Maintain the basic information of the relevant person in charge, including name, contact information (email, DingTalk work number, etc.), support adding, modifying, and deleting the person in charge information, connect to the corporate address book or human resources system, synchronously obtain the latest employee information, and assign a unique identifier to each person in charge to facilitate the call of other modules; 2) Authority control: Different authority levels are assigned to different persons in charge, such as only receiving standard updates of the department or being in charge of the overall affairs. Notifications are sent to authorized persons in a targeted manner according to the scope and importance of the standard updates. Authority inheritance and delegation are supported. For example, the superior person in charge can view all updates within the scope of the subordinates. Authority auditing function is provided to record the authority change history. 3) Interaction interface: Provides a standard API interface to support calls from other modules. The interface includes obtaining a list of responsible persons, querying responsible person information, adding / modifying responsible persons, etc. It provides the notification sending module with the receiving information of the relevant responsible persons (DingTalk ID, email address, etc.), receives the feedback or confirmation information from the responsible persons, and stores it in the database.

Images