Supercharge Your Innovation With Domain-Expert AI Agents!

Data acquisition system, method and device based on message-oriented middleware

A technology of data acquisition system and message middleware, applied in the computer field, can solve the problems of difficulty and lack of data monitoring implementation, and achieve the effect of efficient data acquisition scheme, high stability and guaranteed reliability.

Pending Publication Date: 2021-04-02
MUYUAN FOODS CO LTD
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The premise of data monitoring is data collection. However, there is currently a lack of reliable data collection solutions in a distributed environment, which brings difficulties to the implementation of data monitoring.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data acquisition system, method and device based on message-oriented middleware
  • Data acquisition system, method and device based on message-oriented middleware
  • Data acquisition system, method and device based on message-oriented middleware

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] The core of this application is to provide a data collection system, method and device based on message middleware, so as to realize an efficient data collection process in a distributed environment.

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the drawings and specific implementation methods. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0040] Embodiment 1 of a data acquisition system based on message middleware provided by this application is introduced below.

[0041] Scrapy is an application framework suitable for crawling website data and extract...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data acquisition system based on message-oriented middleware. The system comprises a main node, the message-oriented middleware, a plurality of child nodes and a database, and is characterized in that the main node is used for acquiring a data source URL list and distributing URLs in the data source URL list to the child nodes through the message-oriented middleware; andthe child nodes are realized based on a Scrapy acquisition framework and are used for acquiring data according to the received URLs and storing data acquisition results into a database through a database connection pool. Visibly, the system realizes an efficient data acquisition scheme in a distributed environment by expanding the Scrapy acquisition framework, and each node and the message middleware are relatively high in stability, so that the reliability of the whole data acquisition system can be ensured. In addition, the invention further provides a data acquisition method and device based on the message-oriented middleware, and the technical effect of the data acquisition method and device corresponds to that of the system.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a data collection system, method and device based on message middleware. Background technique [0002] With the development of Internet technology and the surge of data, data monitoring is very necessary in a distributed environment. The premise of data monitoring is data collection. However, at present, there is a lack of reliable data collection solutions in distributed environments, which brings difficulties to the implementation of data monitoring. Contents of the invention [0003] The purpose of this application is to provide a data collection system, method and device based on message middleware to solve the current problem of lack of reliable data collection solutions in distributed environments. The specific plan is as follows: [0004] In the first aspect, the present application provides a data acquisition system based on message middleware, including: ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/951G06F16/955G06F9/54
CPCG06F16/951G06F16/9566G06F9/546G06F2209/547G06F2209/548
Inventor 张玉良陈烨彭佳勇屠宏伟任丽娜聂鑫王平
Owner MUYUAN FOODS CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com