A method and system for transmitting data based on queue

By using a combination of transmission queues and slewing queues in data transmission, the problems of performance loss and memory consumption during data transmission are solved, batch processing of data and container recycling and reuse, and system performance and memory management efficiency are improved.

CN110297719BActive Publication Date: 2024-11-26BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN201810244768.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-03-23
Publication Date
2024-11-26
Estimated Expiration
2038-03-23

AI Technical Summary

Technical Problem

The prior art has problems of performance loss and memory consumption during data transmission, especially when processing data rate differences and data batch processing, the performance impacts of data stacking and frequent creation and recycling of containers in queues.

Method used

By using a combination of transfer queues and slewing queues, the recycling and reuse of data containers is achieved, avoiding the creation of new containers every time they are written, and only locking them once during reading is required to control the amount of data in memory.

Benefits of technology

It realizes batch transmission of data and recycling of containers, reduces the consumption of lock operations, avoids the impact of excessive memory consumption and garbage collection on performance, and improves system performance and memory management efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110297719B_ABST
    Figure CN110297719B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for transmitting data based on a queue, and relates to the field of computer technology. A specific implementation of the method includes: confirming that there is no empty container in the initial container list, then obtaining an empty container from the rotary queue; writing data into the container, and writing the container with written data into the transmission queue; obtaining a container with data from the transmission queue, reading and emptying the data in the container, and writing the empty container into the rotary queue. The present invention can realize batch transmission of data and recycling and reuse of containers. Furthermore, it can avoid creating a new container every time a queue is written, and in object-oriented languages, it can also eliminate the impact of garbage collection on system performance caused by this.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for transmitting data based on a queue. Background Art

[0002] In data processing systems, queues are often used to transmit data between modules. A queue is a first-in-first-out linear table. Figure 1 It is a schematic diagram of a method for transmitting data in the prior art, such as Figure 1 As shown, module A writes data into the queue, and module B reads data from the queue. In this process, the queue acts as a buffer, which can solve the difference in data processing rate between module A and module B to a certain extent. Specifically, module A first obtains the write lock of the queue, and then module A writes data into the queue according to the write lock. It can write one piece of data at a time, or write a batch of data each time through a container. After module A successfully writes the data, it releases the write lock of the queue. In the process of reading data, module B obtains the read lock of the queue, and then module B reads a piece of data from the queue according to the read lock, or obtains a container containing data from the queue according to the read lock. After reading or obtaining successfully, module B releases the read lock. If module A writes data into the queue through a container, it needs to continuously generate new containers. At the same time, module B obtains the container from the queue, and after reading the data in the container, it directly invalidates the container.

[0003] The existing technology has the following problems during implementation: if only one piece of data is processed each time it is written / read, there will be a certain performance loss due to the need to lock each time it is written / read, and it is not friendly to support batch processing of data; if the queue does not limit the length, if the write rate is greater than the read rate, more and more data will accumulate in the queue, which may easily cause memory leakage; if data is written / read through containers, it is necessary to continuously generate containers and invalidate each used container, which will result in a large amount of memory consumption. Summary of the invention

[0004] In view of this, the embodiment of the present invention provides a method and system for transmitting data based on queues, which can store containers with data through a transmission queue and empty containers through a rotation queue. After obtaining a container from the transmission queue, the data in the container is read and emptied, and the emptied container is written into the rotation queue, which can not only realize batch transmission of data, but also realize the recycling and reuse of containers. Furthermore, it can avoid creating a new container every time a queue is written, and in object-oriented languages, it can also eliminate the impact of garbage collection on system performance caused by this.

[0005] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for transmitting data based on a queue is provided.

[0006] The method for transmitting data based on a queue in an embodiment of the present invention includes: confirming that there is no empty container in the initial container list, obtaining an empty container from the rotation queue; writing data into the container, and writing the container with written data into the transmission queue; obtaining a container with data from the transmission queue, reading and emptying the data in the container, and writing the empty container into the rotation queue.

[0007] Optionally, before obtaining the empty container from the turnaround queue, the method further includes: creating an initial container list, wherein the initial container list includes a first number of empty containers; and obtaining the empty container from the initial container list until it is confirmed that there is no empty container in the initial container list.

[0008] Optionally, the first number and the second number are set according to the size of the memory, and the second number is a threshold value of the number of data stored in the container.

[0009] Optionally, the step of writing the container for writing data into the transmission queue includes: acquiring a write lock of the transmission queue; writing the container for writing data into the transmission queue according to the write lock, and releasing the write lock;

[0010] The step of writing the emptied container into the rotary queue includes: acquiring a write lock of the rotary queue; writing the emptied container into the rotary queue according to the write lock, and releasing the write lock.

[0011] To achieve the above objective, according to another aspect of an embodiment of the present invention, a system for transmitting data based on a queue is provided.

[0012] The system for transmitting data based on queues in an embodiment of the present invention includes: a transmission queue, a rotation queue, a writing module, and a reading module; the writing module is used to confirm that there is no empty container in the initial container list, and then obtain an empty container from the rotation queue; write data into the container, and write the container with written data into the transmission queue; the reading module is used to obtain a container with data from the transmission queue, read and clear the data in the container, and write the cleared container into the rotation queue.

[0013] Optionally, the method further comprises a creation module, which is used to create an initial container list, wherein the initial container list includes a first number of empty containers; and obtain empty containers from the initial container list until it is confirmed that there are no empty containers in the initial container list.

[0014] Optionally, the creation module is further used to set the first quantity and the second quantity according to the size of the memory, and the second quantity is a quantity threshold of the data stored in the container.

[0015] Optionally, the write module is also used to write the container of write data into the transmission queue, including the step of: obtaining the write lock of the transmission queue; writing the container of write data into the transmission queue according to the write lock, and releasing the write lock; the read module is also used to write the empty container into the rotation queue, including the step of: obtaining the write lock of the rotation queue; writing the empty container into the rotation queue according to the write lock, and releasing the write lock.

[0016] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device based on queue data transmission is provided.

[0017] An electronic device for transmitting data based on a queue in an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned methods for transmitting data based on a queue.

[0018] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, any of the above methods for transmitting data based on a queue is implemented.

[0019] One embodiment of the above invention has the following advantages or beneficial effects: it is possible to store containers with data through a transmission queue, and to store empty containers through a rotation queue. After obtaining a container from the transmission queue, the data in the container is read and emptied, and the emptied container is written into the rotation queue. Then, a batch of data can be written into the queue through the container each time, which is convenient for batch processing of data. At the same time, when writing / reading a container with a batch of data, only one write / read operation of the queue is performed once, and only one lock is required, which greatly reduces the consumption of locking each data when writing / reading in the existing scheme. Moreover, the number of data written in each container can be set, and the total number of containers can also be set according to the memory. Then, the embodiment of the present invention will not cause excessive accumulation of data in the memory, it has adaptability, and only N*X data will be accumulated at most, where N represents the total number of containers, and X represents the number of data contained in each container. The embodiment of the present invention recycles the container for reuse through the rotation queue, which can avoid creating a new container every time the queue is written. In some object-oriented languages, the impact of garbage collection on system performance caused by this can be eliminated, and the rotary transmission of data is realized.

[0020] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0022] Figure 1 It is a schematic diagram of a method for transmitting data in the prior art;

[0023] Figure 2 is a schematic diagram of the main process of the method for transmitting data based on a queue according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of a method for transmitting data based on a queue according to an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of a method for writing data according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of a method for reading data according to an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of main modules of a device system for transmitting data based on a queue according to an embodiment of the present invention;

[0028] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0029] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] Figure 2 is a schematic diagram of the main process of the method for transmitting data based on a queue according to an embodiment of the present invention, such as Figure 2 As shown, the method for transmitting data based on a queue in an embodiment of the present invention mainly includes:

[0032] Step S201: confirm that there is no empty container in the initial container list, then obtain an empty container from the turnaround queue; write data into the container, and write the container with written data into the transmission queue. Among them, the container is a data structure that can be used to store other objects, and these objects can be combined together. Each container can store more than one piece of data. The turnaround queue is used to store empty containers.

[0033] Before obtaining an empty container from the rotary queue, an initial container list is created, and the initial container list includes a first number of empty containers; and an empty container is obtained from the initial container list until it is confirmed that there is no empty container in the initial container list. Moreover, the first number and the second number can be set according to the size of the memory, and the second number is the threshold value of the number of data stored in the container. In the prior art, if new containers need to be continuously generated when data is transmitted through containers, the total amount of data in the memory cannot be controlled because new containers can be continuously generated. For example, when the queue does not limit the length, since empty containers can be continuously obtained for writing data, if the write rate of the queue is greater than the read rate, more and more data will be accumulated in the queue, which is easy to cause memory leakage. In an embodiment of the present invention, the total amount of data in the memory can be limited by setting the number of containers. If the number of containers is fixed during data transmission, the amount of data that can be loaded in each container is also limited, so the total amount of data in the memory is limited, and the memory occupancy is also limited, thereby avoiding excessive memory consumption.

[0034] Step S202: Get a container with data from the transmission queue, read and empty the data in the container, and write the empty container into the rotation queue. Write the empty container into the rotation queue, and then when an empty container is needed, it can be directly obtained from the rotation queue. This realizes the recycling and reuse of containers, avoids creating a new container every time a queue is written, and eliminates the impact of garbage collection on system performance.

[0035] When getting an empty container, you need to get the write lock of the rotary queue, because it will be deleted from the queue after getting it. When writing a container with data to a transmission queue, the write lock of the transmission queue will be obtained, and the write lock will be released after successful writing. The transmission queue and the rotary queue have their own locks, and they do not affect each other. At present, most language class libraries have ready-made queue class libraries, and you can directly use the queue methods provided by the library. The locks are encapsulated in these libraries, so the application does not need to deal with the locks, but the locks are one of the factors affecting performance.

[0036] In the prior art, if data is transmitted in batches through containers, it is necessary to continuously generate new containers and discard the used containers, resulting in excessive memory consumption; if a single piece of data is processed, it is necessary to lock each time it is written / read, and there is also a certain performance loss. In an embodiment of the present invention, a specified number of empty containers are initialized in the initial container list, and after the empty containers in the initial container list are written with data, the container containing the data is written into the transmission queue. After reading the data of the container in the transmission queue, the used container is emptied and written into the rotation queue, so that the empty container can be directly obtained from the rotation queue. Therefore, the embodiment of the present invention realizes the recycling and reuse of containers through the rotation queue, which can not only transmit data in batches, but also avoid the impact of garbage collection on system performance. At the same time, during the data transmission process, it is only necessary to initialize and create a specified number of containers in the initial container list, without continuously generating new empty containers, which is conducive to improving system performance.

[0037] Figure 3 is a schematic diagram of a method for transmitting data based on a queue according to an embodiment of the present invention, such as Figure 3 As shown in the figure, there are two queues between module A and module B, the transmission queue and the turnaround queue. The transmission queue is used to send containers with data from module A to module B, and the turnaround queue is used to return empty containers from module B to module A. According to the granularity of module A sending data to module B, there is 1 transmission queue and 1 turnaround queue, but if module A sends data to both module B and module C, there will be 1 transmission queue and 1 turnaround queue between module A and each other module.

[0038] When there are no empty containers in the initial container list, module A obtains an empty container from the rotary queue, loads a batch of data into the container, and then writes the container to the transmission queue. After module B obtains a container with data from the rotary queue, it consumes (reads) the data in the container and then empties the container, and writes the emptied container to the rotary queue, so that module A can obtain an empty container from the rotary queue, which not only realizes batch transmission of data, but also realizes the recycling and reuse of containers.

[0039] Figure 4 is a schematic diagram of a method for writing data according to an embodiment of the present invention. Figure 4 As shown, the method for writing data in an embodiment of the present invention includes:

[0040] Step S401: Determine whether there is an empty container in the initial container list. If yes, execute step S405, otherwise execute step S402.

[0041] Before this step, an initial container (such as a java.util.Collection in the Java language) list including a specified number (assuming N) of empty containers is initialized and created. Moreover, the number N of the empty containers can be set according to the size of the memory. After the initial container list is created, when writing data to the queue, the empty containers are first obtained from the initial container list. After all the empty containers in the initial container list are taken out, the empty containers in the rotary queue are obtained.

[0042] Step S402: Get an empty container C from the rotary queue. If the rotary queue is empty, that is, there is no container in the rotary queue, block and wait until there is an empty container in the rotary queue.

[0043] Step S403: Write X data to container C. After obtaining an empty container, write X data to be sent to the container, where the range of X is greater than or equal to 1 and less than or equal to the preset threshold. The preset threshold of X can be set according to the transmission requirements or the size of the memory, for example, set to 100, or set to 200, or set to 5000, etc., as long as it is greater than 0. The more data is installed in the container, the more memory is required. Therefore, if the memory is relatively large, the preset threshold can be set larger, and if the memory is relatively small, the preset threshold can be set smaller. When writing X data to container C, the container can be written in batches or one by one.

[0044] Step S404: Write the container C containing data into the transmission queue. At this time, the container C contains one or more pieces of data, and batch transmission of data can be performed.

[0045] Step S405: Get an empty container C from the initial container list. If there are still empty containers in the initial container list, get an empty container from the initial container list, and execute the above steps S403 and S404 until all containers in the initial container list are read.

[0046] Figure 5 is a schematic diagram of a method for reading data according to an embodiment of the present invention. Figure 5 As shown, the method for reading data in an embodiment of the present invention includes:

[0047] Step S501: Get a container C from the transmission queue. The obtained container contains X pieces of data. If the transmission queue is empty, block and wait until there is a container in the transmission queue.

[0048] Step S502: Read and consume X pieces of data in container C. In the process of reading data, the X pieces of data may be read in batches or one by one, and may also be processed synchronously or asynchronously.

[0049] Step S503: Empty container C and write the container C into the rotation queue. In the process of emptying the container C, the container C can be emptied in batches at one time or one by one.

[0050] According to the embodiment of the present invention, it is possible to store containers with data through the transmission queue, and to store empty containers through the rotation queue. After obtaining the container from the transmission queue, the data in the container is read and emptied, and the emptied container is written into the rotation queue. Then, a batch of data can be written into the queue through the container each time, which is convenient for batch processing of data. At the same time, when writing / reading a container with a batch of data, only one write / read operation of the queue is performed once, and only one lock is required, which greatly reduces the consumption of locking each data when writing / reading in the existing scheme. Moreover, the number of data written in each container can be set, and the total number of containers can also be set according to the memory. Then, the embodiment of the present invention will not cause excessive accumulation of data in the memory, it has adaptability, and only N*X data will be accumulated at most, where N represents the total number of containers, and X represents the number of data contained in each container. The embodiment of the present invention recycles the container for reuse through the rotation queue, which can avoid creating a new container every time the queue is written. In some object-oriented languages, the impact of garbage collection on system performance caused by this can be eliminated, and the rotary transmission of data is realized.

[0051] Figure 6 is a schematic diagram of main modules of a device system for transmitting data based on a queue according to an embodiment of the present invention, such as Figure 6 As shown, a system 600 for transmitting data based on queues according to an embodiment of the present invention includes a transmission queue 601 , a turnaround queue 602 , a writing module 603 and a reading module 604 .

[0052] The transmission queue 601 is used to store containers with data, and the turnaround queue 602 is used to store empty containers.

[0053] The writing module 603 is used to confirm that there is no empty container in the initial container list, then obtain an empty container from the rotation queue; write data into the container, and write the container with written data into the transmission queue. The writing module is also used to write the container with written data into the transmission queue, including: obtaining a write lock of the transmission queue; writing the container with written data into the transmission queue according to the write lock, and releasing the write lock.

[0054] The reading module 604 is used to obtain a container containing data from the transmission queue, read and clear the data in the container, and write the cleared container into the rotary queue. The reading module is also used to write the cleared container into the rotary queue, including: obtaining a write lock of the rotary queue; writing the cleared container into the rotary queue according to the write lock, and releasing the write lock.

[0055] The system for transmitting data based on a queue in an embodiment of the present invention further includes a creation module, which is used to create an initial container list, which includes a first number of empty containers; and obtain empty containers from the initial container list until it is confirmed that there are no empty containers in the initial container list. The creation module is also used to set the first number and the second number according to the size of the memory, and the second number is the number threshold of the data stored in the container.

[0056] According to the embodiment of the present invention, it is possible to store containers with data through the transmission queue, and to store empty containers through the rotation queue. After obtaining the container from the transmission queue, the data in the container is read and emptied, and the emptied container is written into the rotation queue. Then, a batch of data can be written into the queue through the container each time, which is convenient for batch processing of data. At the same time, when writing / reading a container with a batch of data, only one write / read operation of the queue is performed once, and only one lock is required, which greatly reduces the consumption of locking each data when writing / reading in the existing scheme. Moreover, the number of data written in each container can be set, and the total number of containers can also be set according to the memory. Then, the embodiment of the present invention will not cause excessive accumulation of data in the memory, it has adaptability, and only N*X data will be accumulated at most, where N represents the total number of containers, and X represents the number of data contained in each container. The embodiment of the present invention recycles the container for reuse through the rotation queue, which can avoid creating a new container every time the queue is written. In some object-oriented languages, the impact of garbage collection on system performance caused by this can be eliminated, and the rotary transmission of data is realized.

[0057] Figure 7 An exemplary system architecture 700 is shown to which the method for transmitting data based on a queue or the system for transmitting data based on a queue according to an embodiment of the present invention can be applied.

[0058] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0059] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0060] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0061] Server 705 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results to the terminal device.

[0062] It should be noted that the method for transmitting data based on a queue provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the system for transmitting data based on a queue is generally set in the server 705 .

[0063] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0064] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0065] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0066] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0067] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0068] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0069] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0070] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, may be described as: a processor including a transmission queue, a rotation queue, a writing module and a reading module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the writing module may also be described as "confirming that there are no empty containers in the initial container list, obtaining an empty container from the rotation queue; writing data to the container, and writing the container with written data to the module of the transmission queue".

[0071] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0072] According to the embodiment of the present invention, it is possible to store containers with data through the transmission queue, and to store empty containers through the rotation queue. After obtaining the container from the transmission queue, the data in the container is read and emptied, and the emptied container is written into the rotation queue. Then, a batch of data can be written into the queue through the container each time, which is convenient for batch processing of data. At the same time, when writing / reading a container with a batch of data, only one write / read operation of the queue is performed once, and only one lock is required, which greatly reduces the consumption of locking each data when writing / reading in the existing scheme. Moreover, the number of data written in each container can be set, and the total number of containers can also be set according to the memory. Then, the embodiment of the present invention will not cause excessive accumulation of data in the memory, it has adaptability, and only N*X data will be accumulated at most, where N represents the total number of containers, and X represents the number of data contained in each container. The embodiment of the present invention recycles the container for reuse through the rotation queue, which can avoid creating a new container every time the queue is written. In some object-oriented languages, the impact of garbage collection on system performance caused by this can be eliminated, and the rotary transmission of data is realized.

[0073] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for transmitting data based on a queue, characterized in that: include: Creating an initial container list, wherein the initial container list includes a first number of empty containers; obtaining an empty container from the initial container list, writing data into the empty container, and writing the container with written data into a transmission queue, wherein the container is a data structure for storing other objects and combining these objects together, and each of the containers stores more than one piece of data; When confirming that there is no empty container in the initial container list, obtaining an empty container from the rotary queue, writing data into the empty container, and writing the container with written data into the transmission queue; if the rotary queue is empty, blocking and waiting until there is an empty container in the rotary queue; The step of writing the container of written data into the transmission queue specifically includes: acquiring a write lock of the transmission queue; writing the container of written data into the transmission queue according to the write lock, and releasing the write lock; A container containing data is obtained from the transmission queue, the data in the container is read and cleared, and the cleared container is written into the rotary queue, wherein writing the cleared container into the rotary queue specifically comprises: obtaining a write lock of the rotary queue; writing the cleared container into the rotary queue according to the write lock, and releasing the write lock.

2. The method according to claim 1, characterized in that The first number and the second number are set according to the size of the memory, and the second number is a threshold value of the number of data stored in the container.

3. A system for transmitting data based on a queue, characterized in that: It includes a transmission queue, a turnaround queue, a writing module and a reading module; The system further comprises a creation module, configured to create an initial container list, wherein the initial container list comprises a first number of empty containers; The writing module is used to obtain an empty container from the initial container list, write data into the empty container, and write the container with written data into the transmission queue. When confirming that there is no empty container in the initial container list, obtain an empty container from the rotary queue; write data into the empty container, and write the container with written data into the transmission queue; if the rotary queue is empty, block and wait until there is an empty container in the rotary queue, wherein the container is a data structure for storing other objects and combining these objects together, and each of the containers stores more than one piece of data; writing the container with written data into the transmission queue specifically includes: obtaining a write lock of the transmission queue; writing the container with written data into the transmission queue according to the write lock, and releasing the write lock; The reading module is used to obtain a container containing data from the transmission queue, read and clear the data in the container, and write the cleared container into the rotary queue, wherein writing the cleared container into the rotary queue specifically includes: obtaining a write lock of the rotary queue; writing the cleared container into the rotary queue according to the write lock, and releasing the write lock.

4. The system according to claim 3, characterized in that The creation module is further used to set the first quantity and the second quantity according to the size of the memory, and the second quantity is a quantity threshold of the data stored in the container.

5. An electronic device for transmitting data based on a queue, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in claim 1 or 2.

6. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to claim 1 or 2 is implemented.

Citation Information

Patent Citations

  • Player data processing method

    CN102591936A

  • Inter-core communication method realizing data packet zero-copying based on pipelining mode

    CN102761489A

  • Method and system for inserting cache blocks

    CN103154912A

  • Asynchronous file transmission method, apparatus and system

    CN104348805A

  • Method for distributed cloud-side concurrent collection based on Avro RPC transmission

    CN106502802A